The Graduate University for Advanced
Studies (SOKENDAI)

Doctoral Thesis

Study on Accelerator Operation andDiagnostics in Wide Area Environment

Supervisor:

Author:Prof. Kazuro Furukawa

Akito UCHIYAMA

A thesis submitted in fulfillment of the requirements
for the degree of Doctor of Philosophy
in theAccelerator Science
School of High Energy Accelerator Science

August 2014

THE GRADUATE UNIVERSITY FOR ADVANCED STUDIES (SOKENDAI)

Abstract
School of High Energy Accelerator Science
Doctor of Philosophy
Study on Accelerator Operation and Diagnostics in Wide Area
Environment
by Akito UCHIYAMA

Based on personal experience with accelerator operation and maintenance, the author
felt a deep need for remote operation of accelerators in a variety of situations. Currently,
enormous projects, such as the International Linear Collider (ILC), will be launched asa result of the collaborative effort of multiple institutes. After the ILC’s collaborative
commencement, all associates, except for the host country, will need to have methods
for remote maintenance, control, and monitoring of associated devices via Wide Area
Network (WAN) links. For example, a method has to be provided for connecting to thecontrol system network via WAN links from various collaborating institutions. Such remote operation needs to implement components that have concepts different from thoseof daily operation, for example, communication, man-machine user interface (UI), andsystem security. On the other hand, systems for remote operation have been developed by research organizations. Almost all such systems are allocated enough system
resources.
As described above, remote operation is necessary in many situations. However, it is
not always possible to use environments with rich system resources, such as dedicatednetworks, for remote operation. Therefore, in this study, methods regarding UI andsystem security are discussed for remote operation with limited system resources, suchas low-band width networks.
Considering operator interface (OPI) for remote operation, the use of standard protocols, such as Hypertext Transfer Protocol (HTTP), has certain advantages because
system-dependent protocols are not necessary between remote clients and on-site servers.

ii
On the other hand, Web services that provide a UI component to accelerator control
systems have been utilized for various purposes already because such services possess
the advantages of being platform-independent systems, providing software maintenance,and can be developed rapidly.
In this study, the author develops a client system based on WebSocket as a nextgeneration OPI over the Web using Experimental Physics and Industrial Control System(EPICS) Channel Access (CA); WebSocket is a new protocol provided by the Internet
Engineering Task Force (IETF) for Web-based systems. In order to construct a WebSocket server as an EPICS CA client, add-on software for Node.js, called Node-CA, wasdeveloped by the author in C/C++ using the EPICS CA library, which is included in
the EPICS base. As a result of this implementation, WebSocket-based client systems
have become available for remote operation using EPICS in wide area environments.
In addition, from a practical application standpoint, the remote operation of an accelerator via WAN is beset by a number of issues. One such issue is that the accelerator has
both experimental device and radiation generator characteristics. Therefore, any error
in the operation of the remote control system could result in an immediate breakdown.
For these reasons, the author proposes the implementation of an operator intervention
system for remote accelerator diagnostics and support that can obviate any divergences
between the local control room and remote locations.
The purpose of the proposed operator intervention system is to ensure safe remote
output control of EPICS via the WebSocket server. To simply monitor the status of
the accelerator parameters using the WebSocket-based OPI, permission from the on-site
operator is not necessary, provided that authentication has been accomplished througha Secure Sockets Layer (SSL) connection. The WebSocket-based client system is designed such that output control via remote operation always requires the permissionof an on-site accelerator operator who can intervene at any time. The main part ofthe proposed operator intervention system consists of a Process Variable (PV) gateway
provided by EPICS collaboration, a MySQL database, and Web applications. After the
implementation, the on-site accelerator operator can determine the availability of theequipment, and decide whether to grant operation requests from remote users.
Finally, the system that combines WebSocket-based OPI and an operator intervention
system is implemented for the 28 GHz superconducting electron cyclotron resonance ion
source (28GHz SC-ECRIS), which is one of the components in the RIKEN Radioisotope(RI) Beam Factory project, for verification of usefulness. Using this system, the author
confirmed that the output instructions did not reach the EPICS IOC without the on-site
operator’s permission. In addition, remote operation was performed satisfactorily witha network latency of 200 ms, which is approximately the same network latency between

iiiJapan and the USA. Therefore, networking performance tolerance should allow this system to be used over networks provided by almost all Japanese Internet service providers.
Because the system is implemented with RIKEN 28 GHz SC-ECRIS remote operation,
the author confirmed that this new operator intervention system via WebSocket-based
OPIs is a useful method for ensuring the physical security of operational errors.

Acknowledgements
The author would like to thank Dr. Masanori Satoh, Dr. Tsuyoshi Suwada, Dr. Fusashi
Miyahara, Dr. Norihiko Kamikubota, Dr. Takashi Obina, Dr. Tatsuro Nakamura, Ms.
Misaki Komiyama and Mr. Jun-ichi Odagiri, all of whom provided invaluable comments
and warm encouragements.
Special thanks to Mr. Tomonori Ohki, Mr. Toshimitsu Aihara, Mr. Hiromoto Yamauchi, Mr. Kazuyuki Oyamada, and Mr. Masahi Tamura, who were colleagues when
the author was a member of SHI Accelerator Service, Ltd. Without their support forhis shift work, his study would not have materialized by shortness of the development
time.
The author’s heartfelt appreciation goes to Dr. Yoshihide Higurashi and Dr. Takahide
Nakagawa, who provided carefully considered feedback and support for the system implementation in this study.
The author would like to give heartfelt thanks to Mr. Yukito Furukawa at SPring-8 and
Dr. Kay Kasemir at SNS, who provided helpful comments and technical suggestions for
Web-based systems.
The author would also like to express his gratitude to Dr. Osamu Kamigaito and Dr.
Nobuhisa Fukunishi for their moral support.
Finally, the author is deeply grateful to Dr. Kazuro Furukawa and Dr. Noboru Yamamoto, whose comments and suggestions were immeasurably valuable throughout thecourse of his study.

iv

Contents
Abstracti

Acknowledgements

iv

Contentsv

List of Figuresix

List of Tablesxi

1 Introduction
1
1.1 Remote Operation for Large Experimental Physics Facility . . . . . . . . 1
1.2 International Linear Collider . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Global Accelerator Network . . . . . . . . . . . . . . . . . . . . . . 2
1.2.2 GANMVL scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 History of Remote Experiment Systems . . . . . . . . . . . . . . . . . . . 3
1.3.1 Compact Muon Solenoid (CMS) Experiment . . . . . . . . . . . . 3
1.3.2 A Toroidal LHC ApparatuS (ATLAS) Experiment . . . . . . . . . 4
1.3.3 Photon Factory (PF) Experiment at KEK . . . . . . . . . . . . . . 4
1.3.4 Remote Experiment System at SPring-8 . . . . . . . . . . . . . . . 4
1.4 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4.1 Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4.2 Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4.3 Human Communication . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4.4 Man-machine User Interface . . . . . . . . . . . . . . . . . . . . . . 6
1.4.5 System Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Tasks of the Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6 Composition of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Modern Accelerator Control System
12
2.1 Standard Model of the Control System . . . . . . . . . . . . . . . . . . . . 12
2.1.1 Previous Control System . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.2 Developing Control System based on Standard Model . . . . . . . 14
2.2 Control System Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 EPICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.2 DOOCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.3 MADOCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16v

Contents

2.3

2.4

2.2.4 TANGO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Front-end Controller and Device Interface . . . . . . . . . . . . . . . . . .
2.3.1 VME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.2 cPCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.3 PLC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.4 microTCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
EPICS Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 EPICS Runtime Database . . . . . . . . . . . . . . . . . . . . . . .
2.4.2 EPICS CA Protocol . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.3 OPI as Presentation Layer . . . . . . . . . . . . . . . . . . . . . . .
2.4.3.1 MEDM/EDM . . . . . . . . . . . . . . . . . . . . . . . .
2.4.3.2 CSS/BOY . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.3.3 caQtDM . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.4 Channel Archiver . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.5 Alarm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vi
18
19
19
19
19
20
21
22
25
25
26
28
28
28
29

3 Web-based Systems for Accelerator Operation
30
3.1 Advantage of Web-based System . . . . . . . . . . . . . . . . . . . . . . . 30
3.2 Implementation of Web-based System for Static Access . . . . . . . . . . . 32
3.2.1 Hardware Maintenance for GAN . . . . . . . . . . . . . . . . . . . 32
3.2.2 Video Sharing System at KEKB . . . . . . . . . . . . . . . . . . . 32
3.2.3 MyDAQ2 at SPring-8 . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2.4 Electric Log Notebook . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2.5 EPICS Channel Archive Viewer . . . . . . . . . . . . . . . . . . . . 33
3.2.6 NAGIOS Alarm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3 Implementation of Web-based System for Dynamic Access . . . . . . . . . 34
3.3.1 CAML and WebCA . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.3.2 Web Services Interface to EPICS CA . . . . . . . . . . . . . . . . . 35
3.3.3 Flex-based Web Interface . . . . . . . . . . . . . . . . . . . . . . . 37
3.3.4 WebOPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4 Communication in Remote Operation . . . . . . . . . . . . . . . . . . . . 37
3.5 Remote Operation for Troubleshooting . . . . . . . . . . . . . . . . . . . . 38
3.6 Cost-effective Accelerator Operation . . . . . . . . . . . . . . . . . . . . . 39
4 WebSocket Interface for EPICS
40
4.1 Development Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.2 WebSocket Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.2.1 Overview of the Protocol . . . . . . . . . . . . . . . . . . . . . . . 41
4.2.2 WebSocket API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.2.3 Example of WebSocket code . . . . . . . . . . . . . . . . . . . . . . 44
4.3 WebSocket for EPICS-based System . . . . . . . . . . . . . . . . . . . . . 45
4.4 Server-Side System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.5 Channel Access for Node.js . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.5.1 Sever-side caGet Example . . . . . . . . . . . . . . . . . . . . . . . 47
4.5.2 Sever-side caPut Example . . . . . . . . . . . . . . . . . . . . . . . 48
4.5.3 Server-side caMonitor Example . . . . . . . . . . . . . . . . . . . . 48
4.6 WebSocket Server with EPICS CA . . . . . . . . . . . . . . . . . . . . . . 49

Contents
4.7
4.8

vii

Client-Side System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Example of Use of Node-CA . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5 Operator Intervention System
55
5.1 Development Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.2 Security Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.3 System Concept of Operator Intervention System . . . . . . . . . . . . . . 58
5.4 Basic Technology for Operator Intervention System . . . . . . . . . . . . . 60
5.4.1 Virtualization Technology . . . . . . . . . . . . . . . . . . . . . . . 60
5.4.2 EPICS Process Variable Gateway . . . . . . . . . . . . . . . . . . . 60
5.4.3 EPICS Access Security Group . . . . . . . . . . . . . . . . . . . . . 62
5.5 System Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.6 System Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.6.1 Required Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.6.2 Preventing Impersonation . . . . . . . . . . . . . . . . . . . . . . . 64
5.6.3 Setting Upper/Lower Limits . . . . . . . . . . . . . . . . . . . . . . 65
5.6.4 UIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.6.5 Security for WebSocket-side . . . . . . . . . . . . . . . . . . . . . . 67
5.7 Similar System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6 System Implementation
69
6.1 Target Component for the Implementation . . . . . . . . . . . . . . . . . . 69
6.1.1 RI-Beam Factory Project . . . . . . . . . . . . . . . . . . . . . . . 69
6.1.2 Electron Cyclotron Resonance Ion Source . . . . . . . . . . . . . . 70
6.1.3 28GHz SC-ECRIS . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6.2 Implementation Background . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6.3 28GHz SC-ECRIS Control System . . . . . . . . . . . . . . . . . . . . . . 74
6.3.1 Outline of Control System for 28 GHz SC-ECRIS . . . . . . . . . . 74
6.3.2 Client System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.3.3 Network System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6.4 WebSocket-based client system for 28GHz SC-ECRIS Control System . . 78
6.5 Implementation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
7 Discussion
82
7.1 Performance Measurement of WebSocket-based OPI . . . . . . . . . . . . 82
7.1.1 Network Bandwidth Measurement for WebSocket . . . . . . . . . . 83
7.1.2 Network Bandwidth Measurement for AJAX . . . . . . . . . . . . 85
7.1.3 Network Bandwidth Measurement for X Window System . . . . . 86
7.1.4 Measurement Results . . . . . . . . . . . . . . . . . . . . . . . . . 87
7.2 Future Improvements of WebSocket-based OPI . . . . . . . . . . . . . . . 88
7.3 Challenges for the future . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
8 Conclusion

90

A Achievement93
A.1 International Conference . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Contents

viii

A.2 Domestic Conference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
A.3 Award . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
B Source Code95
B.1 Channel Access for Node.js . . . . . . . . . . . . . . . . . . . . . . . . . . 95
B.1.1 nodeca.cc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
B.1.2 wscript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
B.1.3 ca subscription.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
B.2 WebSocket Server for EPICS CA . . . . . . . . . . . . . . . . . . . . . . . 102
B.2.1 app.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
B.3 Example of WebSocket Client for EPICS CA . . . . . . . . . . . . . . . . 114
B.3.1 index.html . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
B.3.2 index.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
B.4 Operator Intervention System . . . . . . . . . . . . . . . . . . . . . . . . . 115
B.4.1 accept.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
B.4.2 conf.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
B.4.3 end.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
B.4.4 request.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
B.4.5 pooling.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
B.4.6 main.php . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

Bibliography

126

List of Figures
1.1

Comparison of connection methods in client applications during remote
operation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

2.1
2.2
2.3
2.4
2.5
2.6
2.7

Schematic layout of the TRISTAN control system. . . . . . . . . . . . . . 13
Block diagram of RARF control system. . . . . . . . . . . . . . . . . . . . 14
Outline of standard model for control system. . . . . . . . . . . . . . . . . 15
Outline of the control system using DOOCS . . . . . . . . . . . . . . . . . 17
Overview of MADOCA-based control system at SPring-8. . . . . . . . . . 17
Overview of the TANGO-based control system at Soleil synchrotrons. . . 18
VME-based front-end controller for magnet power supplies at RIKEN
RIBF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.8 PLC based on Yokogawa FA-M3 under construction at RIKEN Nishina
Center. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.9 microTCA-based LLRF controller installed in Super KEKB. . . . . . . . . 22
2.10 The system configuration diagram for the EPICS IOC, the runtime database,and CA client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.11 A comparison of an EPICS-based system and a non-EPICS-based system. 26
2.12 Behavior of the CA protocol. . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.13 GUI using EDM for RIKEN 28 GHz super conducting ECR ion source
control in RIBF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.14 Example of a CSS/BOY screen. . . . . . . . . . . . . . . . . . . . . . . . . 28
3.1
3.2
3.3
3.4
3.5

Screenshot of MyDAQ2. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Screenshot of Zope-based electric log notebook (Zlog). . . . . . . . . . . .
The system chart of CAML. . . . . . . . . . . . . . . . . . . . . . . . . . .
The example of CAML-based GUI panel displayed on Web browser. . . .
The example of WebOPI-based GUI panel displayed on Web browser. . .

33
34
36
36
38

4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8

Overview of the communication with the WebSocket protocol. . . . . . . .
Detailed WebSocket frame structure provided by RFC6455. . . . . . . . .
Outline of EPICS-based OPI using WebSocket . . . . . . . . . . . . . . .
Overview of NodeCA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
System chart for WebSocket server. . . . . . . . . . . . . . . . . . . . . . .
Screen shot of sample OPI displayed on PC-based Web browser. . . . . .
Screen shot of example OPI displayed on mobile Web browser. . . . . . .
Screen shot of operational panel using Node-CA for KEKB injector Linac.

43
43
46
47
50
52
53
54

5.1
5.2

Procedure for implementation of proposed operator intervention system. . 59
Outline of virtualization technology. . . . . . . . . . . . . . . . . . . . . . 61
ix

List of Figures
5.3
5.4
5.5
5.6
5.7

Outline of PV gateway. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Flowchart illustrating operation of system for remote connection. . . . . .
System chart showing operator intervention system, EPICS IOCs, and
WebSocket server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Network chart of system in VMware virtualized environment. . . . . . . .
User interface for operator intervention system. . . . . . . . . . . . . . . .

Schematic layout for RIKEN RIBF. 28 GHz SC-ECRIS was constructedin the injection line for RIBF at RIKEN. . . . . . . . . . . . . . . . . . .
6.2 Outline of the ECRIS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3 Photograph of the RIKEN 28GHz SC-ECRIS. . . . . . . . . . . . . . . . .
6.4 Section view of RIKEN 28GHz SC-ECRIS. . . . . . . . . . . . . . . . . .
6.5 Outline of 28GHz SC-ECRIS control system. . . . . . . . . . . . . . . . .
6.6 An x-y plotter chart application constructed by EDM. . . . . . . . . . . .
6.7 Wiki page with a screen capture from the 28GHz SC-ECRIS main control
GUI panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.8 Network diagram of Web communication via a reverse proxy server between office network and control system network. . . . . . . . . . . . . . .
6.9 After operation of the 28GHz SC-ECRIS in the control room, the data
are downloaded to office room via Web services for analysis and R&D. . .
6.10 System diagram of WebSocket-based OPI in EPICS-based control system.
6.11 User interface of WebSocket-based OPI. . . . . . . . . . . . . . . . . . . .
6.12 Test environment with another network-emulated WAN. . . . . . . . . . .

x
61
63
65
66
67

6.1

7.1
7.2
7.3
7.4

8.1

Network chart for measuring network bandwidth. . . . . . . . . . . . . . .
Graph of network traffic for WebSocket-based OPI. . . . . . . . . . . . . .
Graph of network traffic for AJAX-based OPI. . . . . . . . . . . . . . . .
Graph of network traffic for OPI constructed with EDM based on X Window System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

70
71
72
73
74
75
76
77
77
78
80
81
83
84
85
86

Remote operation model using WebSocket and operator intervention system in this study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

List of Tables
1.1

To clarify the study’s tasks, a comparison between previous studies andthis study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

92.1Type of records in EPICS runtime database by default. . . . . . . . . . . 23

4.1
4.2
4.3

Types of WebSocket events. . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Types of WebSocket methods. . . . . . . . . . . . . . . . . . . . . . . . . . 44
Web browsers supported by the WebSocket-based OPI as of July, 2012. . 52

5.1
5.2

Physical server specifications. . . . . . . . . . . . . . . . . . . . . . . . . . 65
Comparison of features between new WARCS and proposed operator intervention system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

xiChapter 1

Introduction
1.1

Remote Operation for Large Experimental Physics Facility

Based on an experience with accelerator operation and maintenance, the author felt adeep need for remote operation in a variety of situations. In control system technology,
there have been several studies and attempts at implementing such remote operation.
Through reference to previous works and his own consideration, the author organized
required features in remote operation. Based on the results of a survey, components
with features that are essential to remote operation were implemented through in-depth
technical considerations.
In large experimental physics facilities, such as accelerator facilities or telescope facilities, control systems are operated using applications such as user interfaces (UIs) from acontrol room. In general, control rooms are located approximately less than 10 km fromcontrol systems; in fact, in some accelerator facilities, the control room is located near
the accelerator room. On the other hand, for the maintenance of various equipment,
to perform collaborative physics measurements on different machines, and for several
other reasons, some users demand the ability to conduct remote operations from distant locations, such as out of state, far from the control room. By utilizing modern
control systems with high-performance networks, control system engineers have fulfilled
the demands of some of the features of remote operation.
As part of the history of remote operation, previous studies are described in detail in
the next section.

1

Chapter 1. Introduction

1.2

2International Linear Collider

The International Linear Collider (ILC), which is the next generation accelerator
project, will be launched as a result of the collaborative efforts of multiple institutes
from three regions. The ILC required collaborative effort because the size and cost of
future large experimental physics projects planned for this collider exceed the resourcesof a single region. Even if the development of this accelerator will be divided into
several parts, and technology development will be pushed forward cooperatively, researchorganization will bear responsibility for the project, depending on the development.
Once collaboration has begun, all participating organizations, with the exception of
the host country, must provide methods for remote maintenance, control, and monitoring of their associated devices, because such will be their responsibility. To realize
joint-management of the project from outside the control network, some type of system
implementation for remote operation will be required.
For this purpose, implementation of the global accelerator network (GAN) [1] has beenplanned since March 2000 as a dedicated control system for huge project, such as the
ILC control system.

1.2.1

Global Accelerator Network

The idea underlying GAN is the construction of remote accelerator control rooms that,in terms of performance and feature, are virtually equivalent to the local control room,and can be utilized by remote users of the facility. The technical challenge will more thanlikely be presented by the middle layer software (middleware) that is responsible for thesystem integration, automation, and presentation of the machine state to the accelerator
operator. For GAN study, LabVIEW (produced by National Instruments) was used as
an example of the type of high-level graphical programming tool that was available for
hardware control and integration at Deutsches Elektronen-Synchrotron (German Electron Synchrotron - DESY) in 2001 [2]. To construct a comparatively small-scale control
system, LabVIEW is widely used because of its advantage in software usability, andthe richness of its supported low-level drivers. In the system employed at DESY, theintegration system that uses LabVIEW-based applications can successfully connect tohardware-dependent protocols and to the remote procedure call (RPC) protocol [3].

Chapter 1. Introduction

1.2.2

3

GANMVL scenario

In May 2005, a system similar to GAN, the Global Accelerator Network Multipurpose Virtual Laboratory (GANMVL), was implemented for the European Design Study
towards a Global TeV Linear Collider (EUROTeV) project [4]. GANMVL facilitates
remote operation using a client system that utilizes X11 Windowing, virtual network
computing (VNC), and JAVA VNC [5]. However, these communication methods can encounter issues such as low bandwidth and high network latency during remote operation
over wide area links. Thus, with the limited network resources present in some areas,
if the communication methods described in the previous paragraph are used, stable
operation and control of the accelerator might prove difficult.

1.3

History of Remote Experiment Systems

The technology for remote monitoring already exists with most modern control systems,
and remote operation is generally available with security restrictions from individual
institutions. In this section, the author will provide examples of remote operation ofaccelerator experiments.

1.3.1

Compact Muon Solenoid (CMS) Experiment

Remote operation for this and other experiments that require the Large Hadron Collider (LHC) is achieved through the use of control systems. The researchers in chargeof the Compact Muon Solenoid (CMS) experiment built on the LHC located at theEuropean Organization for Nuclear Research (CERN) in Switzerland, and planned to
install a virtual control room at the Fermi National Accelerator Laboratory (Fermilab)
located just outside Batavia, Illinois [6]. The CMS researchers based at both Fermilab
and CERN have been working closely in the areas of data acquisition, triggers, data
monitoring, data transfer, software analysis, and database access, in order to commission sub-detectors with the common goal of obtaining efficient and high-quality data forphysics study. CMS heavily relies on an online, Web-based monitoring (WBM) systemthat is composed of the Apache Tomcat [7] Web server and the ROOT [8] data display
package. From June through September 2006, the test beam results obtained from the
CMS experiment were made available for online monitoring by anyone via the Web andthe Internet [9].

Chapter 1. Introduction

1.3.2

4

A Toroidal LHC ApparatuS (ATLAS) Experiment

For the ATLAS experiment for the LHC, a remote system was implemented, as is thecase with the CMS experiment described previously [10]. By using the remote monitoring rooms located at American ATLAS institutions for ATLAS experiment, communication ability among American ATLAS members and scientists at CERN was increased.
ATLAS Data Acquisition (DAQ) system can share the information in wide fields, for example of synchronization between processes, error reporting, and operational or physics
event monitoring.

1.3.3

Photon Factory (PF) Experiment at KEK

For the Photon Factory (PF) experiment at the High Energy Accelerator Research
Organization (KEK) in Japan, a special new beamline equipped with facilities to enablea videoconference system, and in some cases, remote operation, was planned by creatinga new “collaboratory” research network [11]. One of the purposes of the collaboratory
research network is to establish a system that allows KEK equipment users to collect datafrom their experiments using remote operation. For this collaborative network, a secure
virtual local area network (VLAN) with authentication of public key infrastructures
is used to connect the computer network between KEK, Tohoku University, Okazaki
National Research Institutes, and Kyoto University [12]. Using Microsoft Windows
Remote Assistance, remote users can share graphical UIs (GUIs) for the control system
located on site [13].

1.3.4

Remote Experiment System at SPring-8

The wide-area remote experiment system (WRES) was developed for safe experimentation through remote operation at SPring-8, a synchrotron radiation facility locatedin Japan [14]. Considering issues of safety for remote experiments, WRES must ensureradiation safety, human physical safety, and network access security. To ensure humanphysical safety during remote operation of the equipment, the physical state of the radiation shield hutch in the experimental equipment is triggered as an “operation-enable
signal” of the interlock system. The first experiment that used WRES was successfully
performed from the RIKEN Wako campus, which is located 480 km away from SPring-8,
in October 2010 [15].

Chapter 1. Introduction

1.4

5

Motivation

To clarify the features required for remote operation, this study is compared to thesystems described in Section 1.2 and 1.3. As a result of such previous works, remote
operation needs to implement components with a concept different from that of daily
operation. Such components are network, middleware, communication, man-machine
UI, and system security. For implementation of the remote operation in this study, thesystem design for these components is discussed based on the ideas and reference ofprevious work. In this section, the components that are required for remote operation
are described in more depth.

1.4.1

Network

For implementation of WRES at SPring-8, the Japanese academic Internet backbone
with a bandwidth of 40 Gbps was used between the experimental equipment at SPring-8
and remote users at the RIKEN Wako campus. Such Internet bandwidth proved to be
sufficient.
As a practical proposition, the GANMVL project and the collaboratory project used
the technology of Virtual Private Networks (VPNs) over Wide Area Networks (WANs)
to enable the connection between remote users and on-site accelerator control systems,
because a dedicated line network would be prohibitively expensive. As a result, VPNs
are effectively utilized as standardized application protocols in the remote operationof control systems. Therefore, VPN needs to be used for construction of the system
environment in this study as well as other remote operation project. On the other hand,
network latency for remote operation from far place, such as between Japan and UnitedStates, is a issue to consider in this study.

1.4.2

Middleware

Modern accelerator control systems have been designed with middleware between hardware devices and client systems in order to integrate hardware-dependent protocols asviewed from the client side. It is possible to improve the efficiency of software development by implementing middleware. For example, in the case of the Experimental
Physics and Industrial Control System (EPICS – one of the major software systems forthe construction of modern control systems), the EPICS input/output controller (IOC),
which is a server that functions as middleware, converts hardware-dependent protocols
to Channel Access (CA) protocol.

Chapter 1. Introduction

6

As shown by EPICS, the solution to the technical problem of integrating GAN-like
control systems is already available, and can be implemented without much difficulty.
For this reason, EPICS is adopted as software for the middleware environment in thisstudy. The middleware features for accelerator control systems are described in Section
2.1.2 in detail.

1.4.3

Human Communication

The sample projects described previously use electronic log notebooks as one of thecommunication tools employed in the remote operation of control systems. In addition,
videoconference systems have been implemented for most remote operation projects inorder to facilitate communication between remote users and on-site staff.
In the case of WRES, remote users can view the status of samples on the on-siteexperiment room using a video-streaming system. As can be intuitively deduced, acommunication tool for on-site staff is an essential feature in the remote operation ofcontrol systems; therefore, different types of video systems are frequently installed forremote operation projects.
On the other hand, human communication between on-site operators and remote users
is also important from a system security point of view because lack of communication
can cause accidents. In addition, Voice over Internet Protocol (VoIP) technology, such
as Skype [16], may play important role. In general, such VoIP-based communication
tools require sufficient network resources.

1.4.4

Man-machine User Interface

For man-machine UIs, any system that interfaces with a GAN project should avoid
dependence on specific platforms employed by GAN projects. The design of a userfriendly UI is essential to ensure the smooth operation of a remote accelerator. Considering operator interfaces (OPIs), there are two main types of methods.
In the first method, OPIs are run on the remote client system by sending messages
to on-site servers based on dedicated protocols (method A). Such a method is a generaldesktop application. The second method exports GUIs to the remote computer using
systems such as the X Window System, Web technologies, and VNC (method B). In
this method, OPI systems use standard protocols to connect the local system (where
the collaborating researchers are located) to the remote computer (where the remoteaccelerator is located), such that the client applications run on the on-site servers. (SeeFig. 1.1)

Chapter 1. Introduction

7

Figure 1.1: Comparison of connection methods in client applications during remote
operation.

There are arguments in favor of the two methods described in the previous paragraphs,
and a mixture of methods is possible. However, in the case of remote operation ofcontrol systems, the use of standard protocols has advantages because system-dependent
protocols are not necessary between the remote client and the on-site server. In mostcases, specialized system-dependent protocols cannot connect through gateways over theInternet.
For the remote operation of GAN projects, the OPI systems are implemented using
traditional methods, such as remote desktop technology. However, these traditional
systems might not operate well over the Internet with low-bandwidth networks. Because
remote desktop technology, such as the X Window System or VNC, is used from the local
computer to remote control desktop environments that can include a Window Manager,a large amount of network resources might be required. Consequently, the availabilityof Web technology for OPIs would be a far more useful method for the remote operationof control systems in low-bandwidth networks, in comparison to remote operation that
relies on remote desktop technology.
However, ergonomic interactivity should be ensured for beam operation. In reality,
traditional Web applications are not suitable for the operation of the accelerator main
parameters, such as Radio Frequency (RF) and magnet power supplies. Thus, WebSocket, a new protocol, is used with EPICS-based control systems in order to utilize
Web technology, such as OPIs, in this study.

Chapter 1. Introduction

1.4.5

8

System Security

For the remote operation of control systems, certain security systems must be considered. From a practical application perspective, the remote operation of an accelerator
via WAN is beset by a number of issues. System security, privacy, interruptions, and network policies are important factors that must be considered in any project that requires
the remote operation of control systems.
The term “system security” includes computer security, as well as the safety of systems,
for example, machine protection, human protection, and human error prevention. Furthermore, communication between on-site operators and remote users is closely related
to system safety for the following reasons:
1. An accelerator has both experimental devices and radiation generator characteristics.
2. An operation error during beam tuning can cause the generation of excessive radiation.
3. Radiation generators can legally be operated remotely from an external location;
however, there are ethical concerns regarding radiation.
4. Generally, an accelerator control system and a system for radiation safety control
are different systems and are separate from each other; therefore, radiation data
might not always be shared with the accelerator control system.
5. All information regarding a facility, including radiation information, can be obtained from accelerator operators in their local control room.
In the case of WRES at SPring-8, the physical status of the radiation shield hutch isused as a signal to trigger the interlocking system of the X-ray beam. On the other hand,researchers for the GANMVL project concluded that a safety system was not necessary
to monitor their control system via remote access from the outside. Note that a safetymechanism is indispensable when the remote operation of control systems must monitor
the parameters of the experiment being performed, and to control the resulting output.
In addition, any errors in the operation of remote control systems could result in
immediate equipment breakdown, and might cause serious problems with human protection without enough communication. To prevent operational errors, the accelerator
conditions must be completely understood by all project participants, particularly by
the on-site operators. Therefore, the author approaches this issue using interposition of
on-site accelerator operators, who can confirm safety conditions.

Chapter 1. Introduction

1.5

9

Tasks of the StudyAs shown in Section 1.4, the components to be improved for remote operation are
the man-machine UI, human communication, and system security. In previous studies,remote control systems have been constructed with an environment based on sufficientcomputer resources. In this thesis, a computer resource means a network-band widthand a dedicated client system. On the other hand, this study aims to implement remote
operation in environments with limited computer resources. Therefore, VoIP-based communication tools (described in Section 1.4.3) that require more network resources are
not covered in this study.
The objective of previous studies and this study for remote operation are summarizedin Table 1.1. The tasks for this study include the possibility of controlling a part of theaccelerator components and to troubleshoot, even with limited computer resources. In
order to manage the tasks, the author believes that use of Web technology is the firstcandidate.
For CMS
Facility
SystemPurposeNeeds ofControl
OutputRequired
SystemResource
PhysicalSecurity
PolicyFor
ATLAS
Experiment Experiment
Data mon- Data monitoring,
itoring andmaintedata analnance,ysis
and data
analysis
No
No

Collaboratry

GANMVL

WRES

This Study

Experiment
Remote Experiment

Accelerator
RemoteAccelerator
Control
Room

Experiment
RemoteExperiment

Accelerator
Troubleshooting
and part ofaccelerator
control

Yes

Yes

Yes

Yes

Many

Many

Many

Huge

Many

Few

None

None

None

Plan Only

UseofInterlock
Systemfor Experimental
hutchOperator
Intervention System

Table 1.1: To clarify the study’s tasks, a comparison between previous studies andthis study.

From the perspective of software usability, Web technology will be one of the mostuseful methods for accelerator control systems. This is because Web-based electronic
log notebooks and JAVA VNC, which is run from a Web browser, have already been implemented to improve the operability of previous remote operation projects. If OPIs arecoded entirely with HyperText Markup Language (HTML) and JavaScript without theChapter 1. Introduction10

use of any remote desktop technology, the system that remotely operates the accelerator
will also have the advantages of Web technology in the remote operation.
Currently, Asynchronous JavaScript and XML (AJAX) technology is widely used forinteractive Web applications, such as Facebook [17] and Google [18]. An AJAX-based
OPI has sufficient functions to monitor numerical values using the Internet, and it canbe used if fast interactive responses are not required. However, when a project requires
remote monitoring of an accelerator in addition to remote operation of said accelerator
with a request for output, it is not realistic to replace GUI-based OPI with Web-based
systems that use AJAX technology, because AJAX technology lacks interactive performance.
To improve the interactive performance of OPIs using the Web, new technology inaccelerator operation must be used to maintain the usability of Web technology. For thisreason, and as a next-generation OPI over the Web that uses the EPICS CA protocol, theauthor proposes a client system based on WebSocket, which is a new protocol provided
by the Internet Engineering Task Force (IETF) for Web-based systems. WebSocket is
Web technology that provides bi-directional, full-duplex communication channels over a
single transmission control protocol (TCP) connection.
Thus, the goals of this study are an examination of the effect of WebSocket-based OPIs
and the implementation of WebSocket-based OPIs in accelerator control systems basedon EPICS, without losing Web usability and with limited computer resources.
In addition, the security issue must be solved if full-scale use of WebSocket-based OPIs
is to become a useful method in the remote operation of control systems. In particular, itis necessary to have exhaustive discussions for system security in remote operation with
output controls. In order to implement safe remote operation, smooth communication
with on-site accelerator operators and physical intervention are required.
To improve the security of remote operations, on-site accelerator operators must know
all the information regarding the devices that are controlled by remote operation. At
the same time, on-site accelerator operators must be able to decide the availability of
remote operation with output devices. In this study, the security of full-scale remotecontrol of accelerators via the Internet is improved, and the usability of such remotecontrol is evaluated.

Chapter 1. Introduction

1.6

11

Composition of the Thesis

In this chapter, the history of remote operation in other facilities was described. Then,the background of this study was presented. In addition, the composition of the thesis
is presented. In Chapter 2, modern accelerator control systems and the standard model
are reviewed as a prerequisite for this study; moreover, the detailed features of the control software are described. In particular, the author further explains the function andbenefits of EPICS. In Chapter 3, the detailed effects and advantages of implementing
the Web-based system for accelerator operation is described. Because Web systems have
many advantages, they have been applied already for various uses in the operation ofaccelerators. In view of previous work, this chapter also clarifies the impact of Webbased systems on accelerator control systems. In Chapter 4, the detailed architecture of
the WebSocket protocol and WebSocket-based system implementation are described. In
addition, because the author designed the WebSocket server to interface to the EPICS
CA protocol, the system software architecture and the implementation of WebSocketbased OPIs as higher-level application programs for remote operation is described. In
this WebSocket-based system, its function and features are also reported. In Chapter 5,
system security for remote operation is discussed. For the security system, the softwarearchitecture as a technical premise in the EPICS-based system is described. Because
physical interposition was necessary as a safety net for remote operation, the required
system and the system design are described in detail. In Chapter 6, system implementation of the WebSocket-based OPIs powered by the security system is described. The
author implemented both systems for 28 GHz superconducting electron cyclotron resonance ion source (SC-ECRIS) at the RIKEN Radioactive Isotope Beam Factory (RIBF)
project. By utilizing a virtual Internet environment using a WAN emulator, this implemented system is tested in a situation similar to remote operation. In Chapter 7, a moregeneral analysis is shown by discussing the methodology of the system design. Several
guidelines to design the control system are presented. In Chapter 8, a summary of thisthesis is described.

Chapter 2Modern Accelerator Control
System
In the past, systems based on minicomputers were used to manage and operate accelerator control systems. In the last 30 years, Ethernet has evolved greatly and has
become extremely popular, not only at home or the office, but also in the industrial
field. The standard model, which has a three-layer structure for control systems, isused widely as the hardware and network standard. Once the standard is established,efficient software can be developed. In addition, the operation of particle accelerators
requires high-performance electronics for beam diagnostics, data acquisition, and machine control. For all the aforementioned reasons, the following control frameworks weredeveloped based on the standard model: EPICS, Distributed Object Oriented Control
System (DOOCS), Message And Database Oriented Control Architecture (MADOCA),
and TAco Next Generation Objects (TANGO).

2.1

Standard Model of the Control System

2.1.1

Previous Control System

In general, large accelerator facilities possess long lifespans, and improvements in theperformance of accelerators are always demanded, even after accelerators become fully
functional. Therefore, control systems must have the flexibility to expand a variety
of functions. In the 1980s, the Transposable Ring Intersecting Storage Accelerator in
Nippon (TRISTAN) control system, which is an electron-positron colliding beam facility
located at KEK, adopted a distributed computer control system using approximately

12

Chapter 2. Modern Accelerator Control System

13twenty 16-bit minicomputers [19]. These computers are linked together by optical fiber
cables to form an N-to-N, token-passing ring network. (See Fig. 2.1)

Figure 2.1: Schematic layout of the TRISTAN control system. (Ref. Computing
in Accelerator Design and Operation Lecture Notes in Physics Volume 215, 1984, pp
367-371, Fig. 1)

Around the same time, the RIKEN Accelerator Research Facility (RARF) control
system consisted of three MELCOM 350-60/500s, which is a 32-bit industrial computer
made by Mitsubishi Electric Corp. The RARF control system also included Computer
Automated Measurement and Control (CAMAC) crate controllers and in-house single
board computers [20]. By using optical fibers, a token-ring type computer network thatconsists of distributed minicomputers was adopted. In the RARF control system, almost
all operations are performed using touch panels. The block diagram of the control system
is shown in Fig. 2.2.
On the other hand, the control system of the Stanford Linear Collider (SLC) is basedon two DEC VAX systems, distributed microprocessor clusters, and CAMAC crate controllers on direct memory access (DMA) that uses a broadband (5—300 MHz) cable TV
system [21]. As described in previous examples, minicomputers were the conventional
method employed for accelerator control systems 30 years ago; on the other, the uniquedevelopment of front-end controllers, hardware, and software were also adopted.

Chapter 2. Modern Accelerator Control System

14

Figure 2.2: Block diagram of RARF control system.(Ref. JAPANESE JOURNAL
OF APPLIED PHYSICS, vol.30, No.11A, November, 1991, pp.2947–2955, Fig. 2)

In the 1990s, accelerator control systems adopted a combination of standard network(TCP/IP, fiber-distributed data interface, Ethernet) and VME-based front-end controllers because standardized networks and hardware were available. In addition, the X
Window System available on UNIX has been widely used as an environment for OPI. On
the other hand, the PC-based control system framework with a two-layer structure (onefor the human UI and the other for the device interface), called the Component Oriented
Accelerator Control Kernel (COACK), was devised by KEK in 1999 [22]. However, thestandard model with a three-layer structure is widely used as the modern accelerator
control system because of its advantages in software reusability, flexibility, and reliability
for distributed control systems.

2.1.2

Developing Control System based on Standard Model

The design concept for control systems was the use of de facto standards such as UNIX,
VME, TCP/IP, and the use of optical Ethernet/IP networks for all device controllers
without any special fields [21]. The standard model is composed of the following threelayers: presentation, equipment control, and device interface. (See Fig. 2.3) At thepresentation layer (for example, the man-machine interface), workstations are used todevelop and run application programs for accelerator operations. This layer also includes a high-speed reliable network. The Input/Output controllers, which are locatedChapter 2. Modern Accelerator Control System

15

within the equipment control layer, are connected to physical hardware through the device interface layer based on various types of field busses. By establishing the standard
model, the research purpose of control systems is shifting from the fundamental control
system design to methods that can share the results of software and hardware for theconstruction of control systems. For this reason, several control system frameworks,
such as EPICS, have been developed using the standard model. The goal of the control system frameworks is to provide the accelerator mechanism as a tool with whichto control processing with flexibility and higher reliability. The main framework for theconstruction of accelerator control systems that is currently being used is introduced in
the next section.

Figure 2.3: Outline of standard model for control system.

Chapter 2. Modern Accelerator Control System

2.2

Control System Framework

2.2.1

EPICS

16

EPICS is major control system toolkit that was initially developed by Los Alamos
National Laboratory (LANL) and Argonne National Laboratory (ANL) for the construction of control systems [23]. Currently, the development of EPICS is progressing atseveral research institutions, such as DESY, Oak Ridge National Laboratory (ORNL),
and KEK, in addition to ANL and LANL, as an international joint development projectcalled EPICS collaboration. In order to integrate hardware-dependent protocols in the
EPICS-based control system, the EPICS Input/Output Controller (IOC), which is aserver middleware that functions as the equipment control layer in the standard model,
is implemented.
Given that the subject of the research in this thesis is an EPICS-based control system,
the EPICS architecture will be described in detail in Section 2.4.

2.2.2

DOOCS

DOOCS is a distributed control system that was developed for the Hadron Electron
Ring Facility (HERA) and the Tera Electron Volt Energy Superconducting Liner Collider (TESLA) Test Facility applications at DESY [24]. DOOCS is an object-oriented
system designed from the device server for integration of front-end controllers to operator consoles. The Equipment Name Server (ENS) consists of a central database that
contains all the names and protocols in the system. In addition, ENS is consulted byclient programs before initial data transfer to a device . The DOOCS system architecture is shown in Fig. 2.4. Communication separates client programs completely fromdevice servers.

2.2.3

MADOCA

MADOCA was developed at SPring-8 in 1995, and has been successfully utilized at
SPring-8 and other facilities, such as HiSOR, NewSUBARU, and SACLA for the control
of accelerator, beam line, and data acquisition system in experiments [25]. The overview
of the MADOCA-based control system is shown in Fig. 2.5. The control commandto the device from the high-level layer application is utilized by sending a message
that is formatted in a human-readable format called SVOC. Currently, MADOCA2, thesuccessor system to the MADOCA system, has been developed [26].

Chapter 2. Modern Accelerator Control System

Figure 2.4: Outline of the control system using DOOCS.
(Ref. http://tesla.desy.de/doocs)

Figure 2.5: Overview of the MADOCA-based control system at SPring-8.
(Ref. http://www.spring8.or.jp/ja/facilities/control_system/madoca)

17

Chapter 2. Modern Accelerator Control System

2.2.4

18

TANGO

TANGO is a control system based on the Common Object Request Broker Architecture (CORBA), and it was developed by the European Synchrotron Radiation Facility(ESRF) and Soleil synchrotrons in France [27, 28]. The TANGO system is used for theconstruction of a computing tool dedicated to the implementation of distributed, heterogeneous, and control/command oriented systems (See Fig. 2.6). CORBA is a definition
for writing object request brokers (ORB). By using TANGO based on CORBA, a given
device is connected as an object via a software bus, and the control actions are executedby invoking methods on objects over ORB. In recent years, a feature to bridge EPICS
and TANGO has been implemented [29].

Figure 2.6: Overview of TANGO-based control system at Soleil synchrotrons.
(Ref. http://www.synchrotron-soleil.fr/)

Chapter 2. Modern Accelerator Control System

2.3

19

Front-end Controller and Device Interface

Computer controls that connect to devices and are located within proximate distanceto the device are called front-end controllers. Front-end controllers communicate with
control system frameworks via the device interface. Recently, various types of networkbased controller that use TCP/IP as the device interface are widely used for control
systems [30]. In general, asynchronous access is required for device interface softwarein this case, because TCP/IP access is a slow line compared with field busses such as
VMEbus. For this purpose, the device support software with asynchronous access has
been provided in the case of EPICS-based systems [31, 32].

2.3.1

VME

VMEbus is a type of computer architecture. The term was first coined in 1980 by thegroup of manufacturers who defined it. Data transfer between modules and the CPU
occurs via the VMEbus. One of the features of the VME-based controller is acceptablereal-time performance. Almost all VME CPU modules support vxWorks as the operating
system (OS). On the other hand, Windows and Linux are also available to run on the
CPU in several cases, for example, the Intel-based (x86) VME CPU module. In thecase of the RIBF control system, VME-based controllers are used to control the power
supply in electric magnets [33]. (See Fig. 2.7)

2.3.2

cPCI

Compact Peripheral Component Interconnected (cPCI) is a standard bus specification
for next-generation industrial computers that has been standardized by the PCI Industrial Computer Manufacturers Group (PICMG). cPCIs are dedicated for fast communication busses and high-performance applications. For example, cPCI is used successfully
for low-level radio frequency control (LLRF) for the Linear Accelerator (LINAC) in the
J-PARC Main Ring injector [34, 35] and for the Korea Superconducting Tokamak Advanced Research (KSTAR) control system [36].

2.3.3

PLC

A Programmable Logic Controller (PLC) (Fig. 2.8) is a digital computer used for the
automation of electromechanical processes, such as the control of machinery on factoryassembly lines, or the elevator. PLCs are used in many industries and machines. Unlike

Chapter 2. Modern Accelerator Control System

20

Figure 2.7: VME-based front-end controller for magnet power supplies at RIKEN
RIBF. The module installed on the left side is VME CPU.

general-purpose computers, PLCs are designed for multiple inputs and output arrangements, extended temperature ranges, immunity to electrical noise, and resistance to vibration and impact. Generally, a development environment called the ladder language isutilized for the development of sequence controls with PLC. PLCs for KEKB, the Japan
Proton Accelerator Research Complex (J-PARC), and the RIBF control system, especially the FA-M3 PLC produced by the Yokogawa electronic corporation, are adopted as
front-end controllers in many situations [37]. Use of the Embedded EPICS technology,
in which EPICS IOC is run inside a PLC CPU module without ladder programming, iswidespread in many cases [38–44].

2.3.4

microTCA

Micro Telecommunications Computing Architecture (microTCA) and Advanced TCA
provide a means for the telecommunications equipment market to take advantage of standardized systems. Recently, microTCA was adopted by DESY and the SuperKEKB as
the next project for the KEKB LLRF control system, as well as for the telecommunications system [45, 46]. In order to achieve higher stability for better beam quality for
Super KEKB, fast digital feedback or feed-forward mechanisms are utilized on LLRF

Chapter 2. Modern Accelerator Control System

21

Figure 2.8: PLC based on Yokogawa FA-M3 under construction at RIKEN Nishina
Center. Two units are connected to each other via FA LINK modules.

through a microTCA module that uses field-programmable gate arrays (FPGAs). There
are several advantages for combining microTCA and EPICS. The EPICS IOC can be
run inside microTCA platforms as an Embedded EPICS technology; that is, the CA
protocol is communicated on the microTCA backplane as a hardware bus. Therefore,system-dependent communication can be managed effectively in a unified manner with
the CA protocol [47]. The picture of microTCA, which is installed for the LLRF of theSuper KEKB project, is shown in Fig. 2.9.

2.4

EPICS Architecture

EPICS software is an excellent representative of modern accelerator control software.
Some of the improvements found in EPICS are described in this work; however, comparedto other available control software, EPICS is reliable and powerful, and it provides asignificant degree of flexibility and performance. EPICS software construction consistsof the following:
EPICS base, a library set for the application programming interface (API) and software
building systems for EPICS application.

Chapter 2. Modern Accelerator Control System

22

Figure 2.9: microTCA-based LLRF controller installed in Super KEKB.

EPICS application, which includes the EPICS runtime database, EPICS IOC, andtools.
EPICS extensions, which are extension applications not included in EPICS base, suchas the EPICS sequencer, AutoSave, and the Motif Editor and Display Manager/Extensible Display Manager (MEDM/EDM).

2.4.1

EPICS Runtime Database

EPICS has adopted a client server model. The system configuration diagram for the
EPICS IOC, the runtime database, and the CA client is shown in Fig. 2.10. By reading
and writing the values in a database called the EPICS Runtime DB through the EPICS
IOC, a control that uses EPICS becomes available [48]. In the EPICS system, a reading
I/O device is described as “Get”, and the action of outputting to I/O devices is describedas “Put”. Unlike relational databases such as PostgreSQL and MySQL, EPICS DB uses
a hierarchical database model. In this model, the type of record is described within therecord itself. For example, when a record works simply by reading analog devices, there
is a line that indicates “ai record” in the database file. The types of records that are
prepared by default in the EPICS system are summarized in Table. 2.1. On the otherChapter 2. Modern Accelerator Control System23

hand, when a record has fields, the behavior of the record is defined by coding the fields.
The following code is an example of an EPICS database file [49].
EPICS record names
aai
aao
ai
ao
aSub
bi
bo
calc
calcout
compressdfanout
event
fanouthistogram
longinlongout
mbbi
mbbiDirect
mbbo
mbboDirect
permissive
sel
seq
state
stringinstringout
subArraysubwaveform

Types
Array Analog Input
Array Analog Output
Analog Input
Analog Output
Array Subroutine
Binary Input
Binary Output
Calculation
Calculation Output
Compression
Data Fanout
Event
FanoutHistogramLong Input
Long Output
Multi-Bit Binary Input
Multi-Bit Binary Input Direct
Multi-Bit Binary Output
Multi-Bit Binary Output Direct
Permissive
SelectSequence
State
String Input
String Output
Sub-Array
Subroutine
Waveform

Table 2.1: Type of records in EPICS runtime database by default.
1

record ( ai , " $ ( user ) : aiExample " )

2

{

3

field ( DESC , " Analog input " )4field ( INP , " $ ( user ) : calcExample . VAL NPP NMS ")5

field ( EGUF , " 10 " )6field ( EGU , " Counts " )7field ( HOPR , " 10 " )8field ( LOPR , " 0 " )9field ( HIHI , " 8 " )10field ( HIGH , " 6 " )11field ( LOW , " 4 " )12field ( LOLO , " 2 " )13field ( HHSV , " MAJOR " )14field ( HSV , MINOR " )15field ( LSV , " MINOR " )16field ( LLSV , " MAJOR " )Chapter 2. Modern Accelerator Control System
17}

Listing 2.1: Example of ai record coded for EPICS database

Figure 2.10: The system configuration diagram for the EPICS IOC, the runtime
database, and CA client

24Chapter 2. Modern Accelerator Control System

2.4.2

25

EPICS CA Protocol

The EPICS CA protocol is implemented in order to enable the communication between
the EPICS server and an EPICS client [50]. A comparison of an EPICS-based system
and a non-EPICS-based system is shown in Fig. 2.11. The CA protocol represents
the presentation layer of the standard model. Through software development in thespecification of the CA protocol, various types of hardware-dependent protocols can be
unified in the client system. The behavior of the CA protocol is shown in Fig. 2.12.
First, the EPICS client searches the records stored in the EPICS server, such as EPICS
IOCs, by sending a user datagram protocol (UDP) broadcast to the subnet. Second,
the EPICS server replies with a UDP packet to the client if the record is found in the
database of the EPICS server. After receiving the UDP packet, the CA client connects
to the EPICS server that sent the UDP packet through TCP. A simple example of a CA
program, which obtains the numerical value from the EPICS IOC and prints the value,
is shown in Listing [49].
1/* caSimpleExamp le . c */2

# include < stddef .h >3# include < stdlib .h >4# include < stdio .h >5# include < string .h >6# include " cadef . h "7main ( int argc , char ** argv ) {8double data ;9chid mychid ;10if ( argc != 2) {11

fprintf ( stderr , " usage : caExample pvname \ n ") ;12

exit (1) ;

13

}

14

SEVCHK ( c a _ t as k_ i ni t i a li z e () ," ca _ t a s k_ in i ti a li z e " ) ;15

SEVCHK ( ca_search ( argv [1] ,& mychid ) ," c a_se a r c h f a i l u r e ") ;16

SEVCHK ( ca_pend_io (5.0) ," ca_pend_io failure " ) ;17

SEVCHK ( ca_get ( DBR_DOUBLE , mychid ,( void *) & data ) ," ca_get failure ") ;18

SEVCHK ( ca_pend_io (5.0) ," ca_pend_io failure " ) ;19

printf ( " % s % f \ n " , argv [1] , data ) ;return (0) ;

20
21

}

Listing 2.2: Simple example of CA client program coded in C

2.4.3

OPI as Presentation Layer

OPI is the software interface between a computer and a human user. In general, OPI isa GUI. In the standard model, OPI is implemented in the presentation layer. Given that

Chapter 2. Modern Accelerator Control System

26

Figure 2.11: A comparison of an EPICS-based system and a non-EPICS-based system. The hardware-dependent protocols are converted by CA protocol in EPICS-based
systems.

GUI development can be costly, the EPICS collaboration project allows several tools foreasy GUI construction .

2.4.3.1

MEDM/EDM

MEDM is a Motif GUI used for designing and implementing control screens [51]. EDM[52], which is another tool for MEDM, also provides the ability to create and edit display
content in the EPICS-based system. All GUI components developed in MEDM/EDM
are displayed on the X Window server. An example of an EDM screen is shown in
Fig. 2.13. Just as drawing a picture is possible using MEDM/EDM, the GUI can beconstructed by arranging GUI components. Currently, the Control System Studio/BestOPI Yet (CSS/BOY) described in the next section is becoming a standard GUI building
tool for EPICS.

Chapter 2. Modern Accelerator Control System

Figure 2.12: Behavior of the CA protocol. (Ref. Part of the EPICS ”Getting Started”
Lecture Series in EPICS Workshop 2006, VECC, India. Original slide is ”Introduction
to Channel Access Clients” by Kenneth Evans, Jr. Semptember 16, 2004)

Figure 2.13: GUI using EDM for RIKEN 28 GHz super conducting ECR ion source
control in RIBF.

27

Chapter 2. Modern Accelerator Control System
2.4.3.2

28

CSS/BOY

In order to integrate the development environment, CSS, which is an Eclipse-based
collection of tools that use JAVA, is provided by the EPICS collaboration project [53].
On the other hand, BOY is implemented as one of the tools included in the CSS forthe construction of OPI [54]. An example of a CSS/BOY screen is shown in Fig. 2.14.
WebOPI (to be described in a later chapter) is also one of the features found in the CSS.

Figure 2.14: Example of a CSS/BOY screen.

2.4.3.3

caQtDM

Instead of the traditional MEDM/EDM based on Motif, a similar function is implemented by caQtDM using Qt [54]. Because Qt is an application UI framework that iscoded in C++, it can be used for developing modern GUIs, such as desktop environments
for major Linux distributions.

2.4.4

Channel Archiver

Data storage is a primary issue in any research facility. Data archiving means moving
data that is no longer actively used to a separate data storage device for long-term
accelerator operation. In the standard model, the archive engine is implemented on thepresentation layer, as well as OPI. For data archiving in EPICS, the Channel ArchiverChapter 2. Modern Accelerator Control System29

can be used as the standard software [55]. On the other hand, the data archiving
function that uses a relational database, such as Oracle database, is implemented in CSS.
Moreover, several data archiving systems developed in-house can also be utilized by eachfacility [56–58]. For example, the proprietary Channel Archiver that uses hadoop, which
is a distributed framework, is developed in-house at J-PARC as a method for effectively
managing big data.

2.4.5

Alarm

The Alarm Handler is provided as standard OPI by EPICS collaboration. By setting
the upper and lower limit in EPICS records, the Alarm Handler sounds an alarm forthe accelerator operator when the set value is over an actual value. In addition to the
Channel Archiver, several in-house alarm systems for EPICS-based control systems are
implemented by each facility [59].

Chapter 3

Web-based Systems for
Accelerator Operation
Currently, the Web is widely used as the most common Internet protocol. Web services
that provide a UI component for accelerator control systems already have been utilized for various purposes because such services have the advantages of being platformindependent systems, providing software maintenance, and can be developed rapidly .
In previous works, Web technology has been utilized to view archived data as an electronic log notebook, and as a monitoring system without fast response. Further, Web
technology is useful as a means of widely informing about the status of accelerators andbeams. For this purpose, WebOPI was implemented by Spartan Neutron Source (SNS)as a Web-based system that uses AJAX with EPICS. By improving Web interactive
performance compared to AJAX, a wide variety of applications, such as those used fortroubleshooting and communication, could become available for the remote operation ofaccelerators because of their advantages. The effect of utilizing Web-based applicationsin accelerator operations is described in this chapter.

3.1

Advantage of Web-based System

Accelerator control systems that use Web services were already devised in 1995 during
development of Internet technologies [60]. Web-based systems have many advantages,
especially in non-large-scale systems. When computers connect to the Internet, suchcomputers become available to any type of information via the Web HTTP is widelyused as the standard protocol through gateways with a firewall built into the network.
In contrast, dedicated protocols cannot easily reach internal hosts because such protocols cannot access the gateway. In addition, most UI requirements might be satisfied
30

Chapter 3. Web-based Systems for Accelerator Operation

31

using “Web service” techniques owing to the rapid development of the Internet andother information networks. Compared to custom-built applications developed using
GUI building toolkits such as JAVA Swing, Web applications have advantages for thefollowing reasons: platform-independence, rapid system development, and ease of system
maintenance.
In traditional systems, the client system is dependent on UNIX and Linux; such systems
also have X Window System servers. In addition, where Microsoft Windows is theplatform, LabVIEW, Visual C, and Visual Basic have been adopted in accelerator control
systems. Then, because of the advent of JAVA, which can run on Mac OS X, Windows,
and Linux, platform-dependency has been resolved to a certain extent. However, JAVA
has not solved completely the issue of platform-independent systems. In the developmentof mobile devices that utilize Android and iOS as the OS, the need to resolve platformdependency has increasingly become an important issue in recent years. These devices
have provided us with many candidates for an operator console, as well as different styles
for the operation of accelerators and for physical experiments. Using a mobile device,accelerator operators can also access other equipment, and can easily troubleshoot thedevices, if required.
JavaScript and HTML provide a simplified, rapid method for creating dynamic Web
content, such as AJAX technology. On the other hand, building GUIs, which are not anessential feature of applications, incurs costs in many cases of desktop applications thatuse common programing languages, such as JAVA, C, and C++. For simple and efficientdevelopment of GUI applications, a combination of HTML and JavaScript enables rapid
development of Web-based applications that are server and platform-independent.
In order to update to the latest version of an application, it is costly and inefficient toprovide a method where a desktop application is required to be installed on many PCs.
From the perspective of software maintenance, the implementation of a Web application
for control systems might result in significantly reduced costs.
In addition, Web-based remote access has possibilities for saving the bandwidth of
network resources compared with traditional technologies, such as VNC, remote desktop,
and the X Window System. Based on these advantages, there have been many attempts
to introduce Web-based applications for accelerator and experimental control.

Chapter 3. Web-based Systems for Accelerator Operation

3.2

32

Implementation of Web-based System for Static Access

From the 1990s to around 2007, almost all Web-based systems have been used forstatic access in control systems, such as applications for archive viewers and log notebooks, where fast response is not required. For comparison, in this section, the author
introduces previous studies that are implementations of Web-based systems for staticaccess.

3.2.1

Hardware Maintenance for GAN

In the GAN project, Web services were discussed to be used for diagnostic access[2]. According to the article, hardware maintenance personnel often have many different
requirements from machine operators; consequently, the Web server interface must allow
the former to perform much of their own software development and maintenance work.

3.2.2

Video Sharing System at KEKB

By using Comet technology, the KEKB control group has constructed a sharing system
in order to view GUI panels and video images using a Web browser [61]. Comet is a Web
application model in which a long-held HTTP request allows a Web server to push data
to a browser. The main feature is to deliver the GUIs, which appear on the X Window
System, to a large number of users using Web technology. By converting the GUI panel
to an image file and providing the image file periodically, the system realizes a display
similar to an interactive action on the Web browser.

3.2.3

MyDAQ2 at SPring-8

At JASRI/SPring-8, a simple data logging and display system, called MyDAQ2, wasdeveloped utilizing a MySQL-based relational database and the Apache HTTP server[62]. For this system, the logged data is displayed and plotted as charts using gnuplot,
and provided by a Comma-Separated Value (CSV) file via a Web browser (See Fig. 3.1).

3.2.4

Electric Log Notebook

Instead of recording information on paper, an electronic log notebook is widely used
as a Web service in accelerator operations [63–66]. Using a Web application for theChapter 3. Web-based Systems for Accelerator Operation

33

Figure 3.1: Screenshot of MyDAQ2.

electronic log notebook is a suitable method, considering their increased flexibility andlower maintenance costs when compared to desktop application electronic notebooks.
The screenshot of the Zope-based electronic log notebook (Zlog), which has been adopted
for KEKB, J-PARC, and RIBF, is shown in Fig. 3.2.

3.2.5

EPICS Channel Archive Viewer

For the KEKB injector linear particle accelerator (LINAC), a Web-based data viewerthat displays and plots data on a chart has been implemented and developed using theChapter 3. Web-based Systems for Accelerator Operation

34

Figure 3.2: Screenshot of Zope-based electric log notebook (Zlog).

JAVA Swing GUI library and a commercial charting library [67].

3.2.6

NAGIOS AlarmAt the Argonne National Laboratory’s Advanced Photon Source, an Infrastructure
Monitoring System (IMS) has been developed to provide immediate notification to theControl Group on-call staff in the event of failure of critical controls hardware components or software processes [68]. In this system, NAGIOS , an open-source host, service,and network monitoring program, has been interfaced to EPICS CA monitoring tools toprovide historical tracking of control infrastructure events, email, and pager notifications
to on-call staff, as well as Internet-accessible status Web displays.

3.3

Implementation of Web-based System for Dynamic Access

After work on Web-based systems with static access, Web-based systems have beenused for monitoring the system of accelerator parameters with dynamic access such as
OPI in accelerator control systems. In this section, the implementation of Web-based
systems with dynamic access by previous studies is discussed because this study also
aims to improve the interactive performance of Web-based EPICS OPI for use withremote operation.

Chapter 3. Web-based Systems for Accelerator Operation

3.3.1

35

CAML and WebCA

As is clear from previous sections, Web applications have been used conventionally forstatic work only, such as viewer applications for archived data and electronic notebooks.
On the other hand, the use of Web applications for static work, in addition to OPIs
with dynamic access, has been implemented since 2008 [69]. In the case of EPICSbased distributed control systems, SNS has planned and implemented the CA Markup
Language (CAML) to call the CA API from an XML file. The behavior of the CA
connection in CAML is based on WebCA, which is the Firefox/Safari plug-in module
developed by Cosylab. Therefore, a Web browser with the WebCA plugin connects to
EPICS IOC using CA directly. The behavior of WebCA, CAML, and a Web browser isshown in Fig. 3.3, and the screenshot of the CAML-based client system is shown in Fig.
3.4.
The main feature of CAML is that the Web browser connects to the CA server directly
via the WebCA plug-in module. The values obtained from the CA server can be managed
in the XML format, and a JavaScript chart library is used for plotting the data.
In CAML-based systems, it is possible to display the GUI using a Web browser. However, there is no use for an HTTP method between the web browser and the CA server,
and the CA protocol is used as dedicated protocol. Though a Web browser is utilized todisplay OPIs, it is not a suitable method for use with remote operation because this system is similar to a general desktop application as described in Section 1.4.5. Given that
CAML utilizes the function of the CA protocol in the layer between the Web browser
and EPICS IOC, the sending of a request might not be reached in the case of remote
operation via a WAN. In addition, CAML depends on the type of Web browser because
it is necessary to use the plug-in for Firefox/Safari.

3.3.2

Web Services Interface to EPICS CA

The standard model of modern accelerator control systems consists of three layers:
the presentation layer, equipment control layer, and device interface layer. In previous
studies [70], a presentation layer on the standard model that is a multi-purpose Webinterface layer has been proposed for the development of higher EPICS CA application
using cross-platform and cross-development methods.

Chapter 3. Web-based Systems for Accelerator Operation

Figure 3.3: The system chart of CAML.

Figure 3.4: The example of CAML-based GUI panel displayed on Web browser.

36

Chapter 3. Web-based Systems for Accelerator Operation

3.3.3

37

Flex-based Web Interface

At Beijing Electron-Positron Collider II (BAPCII), EPICS data that includes beamenergy, beam current, lifetime, and luminosity are displayed in a Web-based real-timechart developed by Adobe Flex [71]. This Flex-based system is utilized by a Web server
installed CAJ, which is a module to interface to JAVA and EPICS CA. On the other
hand, a data acquisition method in this system utilizes polling access to Web servers.

3.3.4

WebOPIAt the International Conference on Accelerator and Large Experimental Physics (ICALEPCS)
2011 in Grenoble, France, the SNS engineers published WebOPI as one of the features
for CSS [72]. WWebOPI aims to provide Web access to EPICS-based OPIs that werecreated in CSS/BOY, without modification. In WebOPI, the function for displaying a
numerical value on the Web in real-time is implemented using AJAX technology. To
publicize information via the Internet, it is only necessary to install Tomcat as the Web
server with JAVA servlet container, and it is not necessary to modify the OPI file createdby CSS/BOY. The screenshot of WebOPI is shown in Fig. 3.5.

3.4

Communication in Remote Operation

It is often necessary to control an accelerator from various locations, in addition to
the central control room, during beam operation and maintenance. On the other hand,accelerator control systems have many types of users: accelerator operators, accelerator engineers, accelerator physicists, and experiment users. During experiments using
the accelerator, many users will need to share standardized displays and be able tocommunicate with each other.
For communication between on-site operators and remote users, it is also necessary
to implement a remote control system that comprises collaboration technologies, suchas text-based chat tools, videoconference tools, and electronic log notebooks. Various
types of communication tools have been developed already, and electronic notebooks arewidely used at standard accelerator control systems.
In the case of GANMVL, video/audio communication tools might be invaluable in allowing experts to work together on accelerator operation at the global level. This isbecause such communication tools help mitigate communication apprehension causedby differences in the native languages of local staff and some of the global experts. According to this model, the expert staff from each laboratory remains based at their home

Chapter 3. Web-based Systems for Accelerator Operation

38

Figure 3.5: The example of WebOPI-based GUI panel displayed on Web browser.

institution, and continues to participate in the machine operation after construction. For
this reason, there is significant utility in implementing such a feature as communication
tools using the Web.

3.5

Remote Operation for Troubleshooting

In general, an accelerator facility consists of many components, such as RF, the magnet’s power supply, vacuum, beam diagnostic instruments, water cooling, and the control
system. Given that the staff/developers of the accelerator components have a responsibility for resolving machine problems, such personnel should verify conditions as soon
as possible if there is a warning sign.
In some rare situations, such as accelerator problems and change of beam conditions(beam emittance, etc.), on-site accelerator operators might require seeking direction
from other engineers or scientists over the telephone when said engineers or scientists areChapter 3. Web-based Systems for Accelerator Operation

39

away from the laboratory. In such scenarios, the ability to provide the exact directions
required by accelerator operators is contingent on the amount of information required
for remote troubleshooting. However, even in cases where the native languages are
not in conflict, it is difficult to correctly provide detailed information about accelerator
conditions via telephone and voice communication, because insufficient information is
obtained by listening compared to observing. For these reasons, there is a significant
need to monitor and maintain devices from anywhere via the Internet and the Web.

3.6

Cost-effective Accelerator Operation

Future accelerator projects will grow larger, resulting in higher electricity costs. Even
if there is only one problem with the components, it might not be possible to provide
beam service to experiment users. In Japan, which is one of the sites proposed for
ILC, electricity costs are increasing because of the Fukushima nuclear accident that
occurred after the Great East Japan Earthquake of 2011. In order to save electricity
costs, possible errors caused by the operation of remote users must be prevented during
troubleshooting tasks performed via remote operation.
For such cases, it is important to reduce beam tuning time and accelerator standby timeto the maximum. Therefore, in this situation, communication between on-site operatorsand remote users is considered extremely important, and all information about theactivities that occur in the local control room should be communicated to remote users.
As described in the previous sections, Web technology is used for applications withdynamic access such as OPI; however, interactive performance is not sufficient. In
addition, given that general accelerator control systems are implemented with dedicatedlocal networks, the control systems must also improve the safety of remote operationand network security when remote users gain access from the Internet.

Chapter 4

WebSocket Interface for EPICS
During beam operation and maintenance, it is often necessary to control acceleratorsfrom different locations and from the central control room. However, it is not practical toreplace the GUI-based OPI with a Web-based system using AJAX technology because ofinteractive performance issues. Therefore, as a next-generation OPI over the Web using
the EPICS CA protocol, the author developed a client system based on WebSocket,
which is a new protocol provided by IETF for Web-based systems. WebSocket is a web
technology that provides bidirectional, full-duplex communication channels over a single
TCP connection. By utilizing Node.js and the WebSocket access library called Socket.IO,
WebSocket servers have been implemented for use in accelerator control systems. Node.js
is a server-side JavaScript language built on the Google V8 JavaScript Engine. In order
to construct a WebSocket server as an EPICS CA client, an add-on for Node.js wasdeveloped by the author in C/C++ using the EPICS CA library, which is included in
the EPICS base. As a result, for accelerator operation, Web-based client systems havebecome available with various types of equipment as well as in central control rooms.

4.1

Development Background

One of the advantages of an EPICS-based system is the unified communication protocol
between a front-end controller and the client systems provided by the CA protocol. Thus,
even when various types of controllers are used as control systems, all client systems, suchas OPIs, can be developed based on the CA protocol without hardware dependencies.
In general, a method that uses the CA libraries or the display manager supported by
EPICS collaboration, such as MEDM/EDM (see Section 2.4.3.1) and CSS/BOY (seeSection 2.4.3.2) has been adopted for GUI development in the EPICS-based system.
On the other hand, the engineers at SPring-8 proposed development methods for the40

Chapter 4. WebSocket Interface for EPICS

41

main OPI using WebSocket and implemented a prototype [73]. The prototype system
was constructed using a MADOCA control system framework that was developed at
SPring-8.
As described in Chapter 3, Web-based systems have many advantages for accelerator
control systems. By utilizing WebSocket technology, it is possible to solve the issuesof Web-based systems, such as those described in Section 3.4, because of dramatic improvements in interactive performance compared with AJAX-based Web applications.
That is, for OPIs, the communication method, and the troubleshooting method, vast
improvements are possible through the Web when remotely operating accelerator equipment. For the EPICS-based system, traditional Web applications lack the interactivity
required to operate some of the hardware for accelerator control systems.
However, in the case of SPring-8, the control system framework is MADOCA, which
is described in Section 2.2.3. Given that real-time Web applications have many advantages, the author developed a corresponding WebSocket server for the EPICS CA, andimplemented Web applications using WebSocket for the EPICS-based control system.

4.2

WebSocket Architecture

WebSocket is a new protocol for achieving bidirectional communication between a WebSocket server and a Web browser. Originally, WebSocket was part of HTML5. Throughdraft versioning, WebSocket was formulated as an RFC6455 by ITEF in December 2011[74]. WebSocket can be used for real-time applications because of low protocol overhead.
The WebSocket API specifies how to create and access WebSocket within a Web browser.
A WebSocket connection is established via JavaScript, and then, it becomes available
for sending/receiving messages and executing error handling through a connection.

4.2.1

Overview of the Protocol

The WebSocket protocol is an independent TCP-based protocol. Its only relationshipto HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.
An overview of the communication for the WebSocket protocol is shown in Fig. 4.1.
The WebSocket specification is as follows:

1. The initial handshake starts at the HTTP level. A Web browser sends a handshake
request to the HTTP server to connect to WebSocket. The following is an example
of a handshake request from the client [74].

Chapter 4. WebSocket Interface for EPICS

42

GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

2. The server sends a handshake response after approving the request from the Webbrowser. The following is an example of a handshake response from the server.
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Protocol: chat

3. After establishing the handshake, the protocol switches to WebSocket, and thena bidirectional communication occurs between the WebSocket server and the Webbrowser.

A more detailed view of the WebSocket frame structure is provided in Fig. 4.2. As
shown in the figure, WebSocket allows interactive responses, thus compensating for
disadvantages that cannot be eliminated with traditional HTML.

4.2.2

WebSocket API

To establish a WebSocket connection with a Web server, it is necessary to send datafrom the Web browser using the send( ) method to the client-side JavaScript. The
following line of code is the API that creates a new WebSocket object.

var Socket = new WebSocket(url, [protocol] );where the first argument, url, specifies the uniform source locator (URL) to which toconnect. In general URL naming convention, the WebSocket protocol defines the ws://
and the wss:// prefixes as indicators for a WebSocket and a WebSocket Secure connection, respectively. The second attribute, protocol, is optional and if present, it specifies
a sub-protocol that the server must support for the connection to be successful. TableChapter 4. WebSocket Interface for EPICS

43

Figure 4.1: Overview of the communication with the WebSocket protocol.

Figure 4.2:

Detailed WebSocket frame structure provided by RFC6455.
https://tools.ietf.org/html/rfc6455)(Ref.

Chapter 4. WebSocket Interface for EPICS
Event
open
message
error
close

Event Handler
Socket.onopen
Socket.onmessage
Socket.onerror
Socket.onclose

44

PurposeSocket connection is established.
Client receives data from server.
Error has occurred with communication.
Socket connection is closed.

Table 4.1: Types of WebSocket events.

4.1 lists the events associated with a WebSocket object. In a WebSocket connection,
data are sent as events from the WebSocket server to the client.
Table 4.2 lists the methods associated with a WebSocket object.
MethodSocket.send()
Socket.close()

Purposesend(data) method transmits data via the connection.
close() method is called before any existing
connection can terminate.
Table 4.2: Types of WebSocket methods.

4.2.3

Example of WebSocket code

In this section, the author shows an example of WebSocket code for client-side JavaScript(see Section 4.1). After the HTTP handshake, the WebSocket starts to connect withthe bidirectional TCP socket between the Web browser and the WebSocket server. This
sample JavaScript code uses the free echo server (http://www.websocket.org/echo.html)
to establish the WebSocket connection.

1

var Socket = new WebSocket ( ’ ws :// echo . websocket . org ’) ;2

Socket . onopen = function () {3

console . log ( " Connection is successful ") ;4

Socket . send ( " Hello world " ) ;

5

};

6

Socket . onmessage = function ( rs ) {7

var receivedData = rs . data ;8

console . log ( " Message exists . " ) ;console . log ( receivedData ) ;910// Hello world};

Listing 4.1: Example of clent-side JavaScript code for echo server.

Chapter 4. WebSocket Interface for EPICS

4.3

45

WebSocket for EPICS-based System

Thus far, various types of Web services in EPICS-based systems have been implemented,
for example, data archive viewer, electric log notebook, and so on (see Section 3.2).
However, such services have been implemented only in Web-based systems, where a fast
response and monitoring are unnecessary, such as for archive viewers and electric-log
systems. This is because it is difficult for a Web-based OPI to realize a real-time response
similar to native applications, such as EDM/MEDM/CSS, even if AJAX technology isutilized. Given that bidirectional transmission between a server and clients is now
possible, the aforementioned problem can be solved with WebSocket. In practice, theauthor has confirmed a sufficient interactive response in a 100-ms cycle using WebSocket
on a Web browser (Mozilla Firefox), and the performance is similar to that of native
EPICS applications. Furthermore, given that periodic polling is unnecessary, unlike
AJAX and Comet, network traffic can be reduced.
OOn the other hand, WebSocket cannot work with Internet Explorer (IE) 9, which is oneof the main browsers in existence today, because a WebSocket connection is unavailable
in IE 9. However, this problem was resolved in IE 10, because Microsoft intended for IE
to be compliant with WebSocket [75]. Currently, IE 10 and higher support WebSocket.
In order to implement a WebSocket-based system with EPICS, the former must consistof a WebSocket server that corresponds with the EPICS CA protocol as the server-sidesystem, and a WebSocket client that is coded based on the document object model(DOM)[76]. The outline of the EPICS-based OPI using WebSocket is shown in Fig. 4.3.

4.4

Server-Side System

Because the development of an entirely original EPICS-based WebSocket server is expensive, the author decided to use a library that includes a WebSocket server function.
Initially, the author made use of libwebsockets [77] and Jetty [78], which also provides a
WebSocket server feature, for the development of WebSocket support for EPICS. However, after considering the difficulty of developing such a WebSocket, the author utilized
Node.js [79] and the WebSocket library Socket.IO [80] to aid in the development of
the WebSocket server. Node.js is one of the server-side JavaScript languages developed
based on the Google V8 JavaScript Engine [81].
One of the main features of Node.js is that it works asynchronously with single-thread
processing. A significant number of human resources might be required for the development of an entirely original asynchronous WebSocket server; however, Socket.IO, which

Chapter 4. WebSocket Interface for EPICS

46

Figure 4.3: Outline of EPICS-based OPI using WebSocket

is a library for the construction of WebSocket servers, is used because it contains all thefunctions required for the development of an asynchronous WebSocket server. Therefore, Node.js 0.6.18 and Socket.IO 0.9.6, which are the stable versions of these tools,
were utilized for the development of the WebSocket server.

Chapter 4. WebSocket Interface for EPICS

4.5

47

Channel Access for Node.js

In order to implement the WebSocket server with a CA connection using Node.js,
Node.js must call the CA API. Therefore, the author developed the add-on software
called NodeCA using C++ in order to interface with the CA from Node.js. Given that
Node.js can call NodeCA, NodeCA utilized the CA library provided by the EPICS base.
To call CA from Node.js, caGet, caPut, and caMonitor, which are basic functions in
EPICS, were created.
In general, an event-driven caMonitor requires a non-blocking algorithm in the programthat uses it, but the single-threading Node.js can cause the thread to be blocked becauseof negligent implementation. Consequently, in order to prevent thread blocking, NodeCA
waits for the CA event in another thread; when the CA event arrives at that thread,
NodeCA sends the message queue to the main thread. Fig. 4.4 shows an overview of
NodeCA. In Fig. 4.4, the software developed by this research is shown in the grey area.
Examples that use NodeCA with caGet, caPut, and caMonitor are provided in thissection. Note that at the start of each example, it is necessary to set require(”nodeca”)
as the add-on software.

Figure 4.4: Overview of NodeCA.

4.5.1

Sever-side caGet Example

This program fetches a value as a TEST:calc record from the EPICS IOC in intervalsof 100 ms using caGet, and outputs the numerical value using console.log( ).
Based on a comparison with a simple CA client written in the C language, which is
described in Section 2.4.2, it is evident that coding the CA client using NodeCA can be
performed with ease.

Chapter 4. WebSocket Interface for EPICS

1

var pv = require ( " ../ build / Release / nodeca " )2function cagets () {48

console . log ( pv . caget ( " TEST : calc " ) ) ;

3
4

}

5

setInterval ( cagets ,100) ;Listing 4.2: Example of server-side JavaScript code for caGet function.

4.5.2

Sever-side caPut Example

This program changes the value that is stored in the TEST:subExample record of EPICS
IOC to five via the WebSocket server.
1

var pv = require ( " ../ build / Release / nodeca " )2

pv . caput ( " TEST : subExample " ,5) ;Listing 4.3: Example of Server-side JavaScript code for caPut function.

4.5.3

Server-side caMonitor Example

This program outputs the numerical value of the TEST:ai record through an eventdriven behavior. (When the TEST:ai record is changed, the value is sent to the client
automatically.) In order to receive the value, the pv.callback method is required before
running the pv.camonitor method, because Node.js is implemented through an asynchronous algorithm.

1

var pv = require ( ’ ../ build / Release / nodeca ’) ;2

function mon () {pv . callback = function ( v ) {3

console . log ( ’ Value : ’ + v ) ;

4

}

5

pv . camonitor ( " TEST : ai " ,0) ;

6
7

}

8

setTimeout ( function () {910

mon () ;} ,1000) ;Listing 4.4: Example of server-side JavaScript code for caMonitor function.

Chapter 4. WebSocket Interface for EPICS

4.6

49

WebSocket Server with EPICS CA

The WebSocket server that corresponds to the EPICS CA protocol was developed
utilizing NodeCA as described in Section 4.3.2. Given that the WebSocket server was
written with Node.js, both the Web server and the Web client are developed using
the JavaScript language. (Recall that Node.js is a server-side JavaScript language.)
The system chart for the WebSocket server is shown in Fig. 4.5. The corresponding
algorithm is described as follows:
1. In the code of the client-side JavaScript, custom events (caGet, caPut, and caMonitor) are sent to the server using the WebSocket protocol.
2. In the case of a caGet/caPut event, the main process makes threads (caGet, caPut),and the threads connect to EPICS IOC using the CA protocol.
3. The information between the main process and the thread is communicated usinga message queue as the interprocess communication (IPC).
4. In the case of a caMonitor event, the main process makes a child process and thechild process makes a thread (caMonitor). The thread continues to connect to
EPICS IOC using the CA protocol.
5. Similar to the case of caGet, in the case of caMonitor, the information between
the main process and the thread is communicated by message queue, and the mainprocess and the child process communicate using the IPC (child.fork( )) provided
by Node.js.

Chapter 4. WebSocket Interface for EPICS

Figure 4.5: System chart for WebSocket server.

50

Chapter 4. WebSocket Interface for EPICS

4.7

51

Client-Side System

An example of coding in the client-side JavaScript is as follows:1

socket . emit ( ’ caMonitor ’ ,{2

pv : ’ Example : ai1 ’ ,data_name : ’ TEST2 ’ ,

3
4

}) ;

5

socket . on ( ’ ca ’ , function ( data ) {67

document . getElementById ( " TEST_record1 ") . innerHTML = data . TEST2 ;}) ;Listing 4.5: Example of caMonitor for client-side JavaScript.

The behavior of this example is as follows:
1. The client-side JavaScript obtains a value from the Example:ai1 record by sendinga message of the event-driven caMonitor in WebSocket.
2. The value is stored in data.TEST2.
3. Through DOM operation, the value is updated on the HTML with respect to eachCA event.
It is possible to realize a text update, such as EDM, by coding DOM with JavaScript on
the Web. DOM is a platform and language-neutral interface that allows programs andscripts to access and update content dynamically, and to structure and style documents[82]. The following is a concrete example of HTML, where a DOM element, such as the<div> tag, is updated on a Web browser through JavaScript code.
1< div id = " hoge " > hogehoge </ div >2< script type = " text / javascript " >34document . getElementById ( " hoge ") . innerHTML = " hello world " ;</ script >Listing 4.6: Example of conding DOM

In this example, the code replaces ”hogehoge”, which is surrounded by a <div> tag,with ”hello world”. This method is used for WebSocket-based systems and for traditional HTML technology, such as AJAX. The difference between AJAX or Comet and
WebSocket is based on the communication method used to update values.
Other libraries used in this WebSocket-based system are flot [83] and jsgause [84],which are jQuery-based JavaScript libraries; these libraries are used in this system forthe visualization of accelerator parameters. Many JavaScript libraries, other than thosementioned at the start of this paragraph, are also available for the visualization ofChapter 4. WebSocket Interface for EPICS

52

information, such as strip charts. As a result, the client system has the advantage oflow development costs because money can be saved in multiple steps.
A verification system was constructed that is useful for remote operation because a
response comparable to native EPICS applications is confirmed. In addition to PCbased Web browsers, the use of OPIs on Web browsers for iPhone4S and Android mobile
is available. A screenshot of WebSocket-based OPIs is shown in Fig. 4.6, 4.7 and the
Web browsers supported by this system as of July 2012 are listed in Table 4.3. Chapter
6, which describes the system implementation, reports such systems in detail.

Figure 4.6: Screen shot of sample OPI displayed on PC-based Web browser (Google
Chrome).

Windows/Linux
Android 4.0
iOS

Google Chrome 20, Mozilla Firefox 14, Opera
11
Google Chrome 18, Mozilla Firefox 14
Safari 5, Google Chrome 19, Mercury

Table 4.3: Web browsers supported by the WebSocket-based OPI as of July, 2012.

Chapter 4. WebSocket Interface for EPICS

53

Figure 4.7: Screen shot of example OPI displayed on mobile Web browsers. Left
figure is for android mobile and right figure is for iPhone4S.

4.8

Example of Use of Node-CA

As well as this study, Node-CA was used for construction of WebSocket-based system
in KEKB Linac control system [85]. In the KEKB injector Linac, information regarding
accelerator operation has been provided through HTTP services and CGI since 1994.
In order to provide such information in real time, WebSocket-based OPI was developed
using Node-CA. As a result, detailed information on the accelerator operation was successfully provided for the entire KEK campus using a new GUI panel, which appears on
the Web (See Fig. 4.8).

Chapter 4. WebSocket Interface for EPICS

Figure 4.8: Screen shot of operational panel using Node-CA for KEKB injector Linac.

54

Chapter 5

Operator Intervention System
Accelerator facility management through international collaboration often requires
methods for remote maintenance, control, and monitoring of associated devices. For
example, a method needs to be provided to connect to the control system network via
WAN links from various collaborating institutions. However, from a practical application standpoint, the remote operation of an accelerator via WAN is beset by a numberof concerns, including security concerns.
A particular concern is that the accelerator has both experimental devices and radiation
generator characteristics. Additionally, any error in the operation of remote control
systems could result in an immediate breakdown. For these reasons, the author proposes
the implementation of an operator intervening system for remote accelerator diagnostics
and support that can obviate any differences between the local control room and remotelocations. After the implementation, the on-site accelerator operator can determine
the availability of the equipment, and decide whether to grant operation requests fromremote users.

5.1

Development BackgroundAs described in Section 1.3, there are several cases of cooperation between experts
and remote operators across WAN links for accelerator operations. In situations suchas accelerator problems and change-of-beam conditions (for example, beam emittancefrom ion sources), on-site accelerator operators require directions over the telephonefrom other engineers or scientists with regard to accelerator components, such as RF,
vacuum, beam diagnostic, control system, and ion source. In such scenarios, the ability
to provide the exact directions that are required by accelerator operators is contingent on
55

Chapter 5. Operator Intervention System56

the amount of information required for remote troubleshooting. Even in cases where thenative languages are not in conflict, it is difficult to correctly provide detailed information
about accelerator conditions via the telephone and through voice communication because
not enough information can be obtained by listening compared with seeing.
On the other hand, and as described in Chapter 4, WebSocket-based operator interfaces(OPIs) are available to resolve problems via WAN links because of the development of a
global Internet society. In addition to monitoring accelerator parameters from locations
distant to the accelerator control room, remote users can operate accelerator devices thatprovide outputs, even if there are no concerns such as radiation safety, human physicalsafety, and network access security.
However, accelerators should be controlled with utmost care because accelerators haveboth experimental devices and radiation generator characteristics. From past experience
with the operation of accelerators, the author believes that the atmosphere of the accelerator control room is an important factor during beam tuning. For example, because
almost all accelerator information is integrated into the accelerator control room, theimplementation of joint monitoring with multiple operators to observe the condition of
the accelerator is a useful method for avoiding any problems that might arise.
In order to prevent operational mistakes, the accelerator conditions need to be understood in their entirety. Therefore, it is necessary to consider a safety system with on-siteaccelerator operator intervention in order to control output via remote access. Note thata safety mechanism is indispensable when the remote operation needs to monitor parameters and control output in order to prevent accelerator problems caused by operational
mistakes.
To address safety issues and improve the usability of remote operations, the author
developed an operator intervention system for WebSocket-based OPIs for the EPICS
[86].

5.2

Security PolicyFirst, the author considered the following in order to protect accelerator networks fromoutside intrusion. In the case of accessing accelerator networks from the WAN, theauthor ensured the security of WebSocket access using VPN and authentication via a
Secure Sockets Layer (SSL). In addition, accelerator operators have to completely understand users attempting to access the network (login ID and so on) and access route (fulldomain of Internet providers and so on) before WebSocket control is allowed. Therefore,
all accessing logs for WebSocket control should be open to accelerator operators.

Chapter 5. Operator Intervention System

57

Next, the author considered instructions for accelerator operation to EPICS IOC fromthe Web browser via WebSocket. For caPut, which provides instructions to each device
or component during accelerator operation, accelerator operators need to know the action and behavior perfectly. On the other hand, it would not matter if such operators
did not understand some simple instructions in detail, such as caGet and caMonitor,
which are used to monitor or obtain numerical values for accelerator parameters. If someaccelerator parameters are changed using the Web application from outside internal networks without considering beam tuning, the accelerator condition might be complicatedin many cases. Thus, when using caPut, a policy is required between accelerator operators in the control room and remote users exerting control remotely over the Web. The
following are proposed for this policy.

1. Before remote users exercise remote control over the Web, they have to communicate with accelerator operators using VoIP system such as Skype.
2. On-site accelerator operators check the user ID and the domain or IP address of
the Internet provider for remote users.
3. Remote users send a request to accelerator operators when sending caPut instructions to the EPICS record.
4. In response to requests from remote users, accelerator operators make judgment
decisions regarding whether components in the EPICS record can be controlledusing WebSocket. A response is returned to remote users regarding whether it ispossible to operate said components in the EPICS record.
5. After receiving this response, remote users can not only monitor, but also operatecomponents via WebSocket.
6. If accelerator operators make judgment decisions to interrupt operation, accelerator operators discontinue WebSocket operations as soon as possible. In addition,
if a certain period passes, permission for WebSocket operation will be cancelled
through timeout.

Chapter 5. Operator Intervention System

5.3

58

System Concept of Operator Intervention System

The purpose of the proposed operator intervention system is to ensure safe remote
output control of EPICS via the WebSocket server. To simply monitor the status of
the accelerator parameters using the WebSocket-based OPI, permission from the on-site
operator is not necessary, provided that authentication has been accomplished throughan SSL connection. The WebSocket-based client system is designed such that output
control via remote operation always requires the permission of an on-site accelerator
operator who can intervene at any time.
The main part of the proposed operator intervention system consists of a Process
Variable (PV) gateway provided by EPICS collaboration [87], a MySQL database, and
AJAX web applications. Unlike OPI, real-time interactive response is not required forsending output control permission, or for viewing access logs in the operator intervention
system. Consequently, in consideration of the difficulty of system development, theauthor utilized a Linux, Apache, MySQL, PHP (LAMP) environment for the operator
intervention system without using WebSocket. The procedure for the implementation of
the proposed operator intervention system is as follows (see Fig. 5.1 for an illustration.).
The detailed system design is described in Section 5.5.

1. First, on-site accelerator operators grant permission to every output of PVs that
corresponds to the EPICS hardware devices.
2. If PVs have not been granted permission, the operator intervention system always
rejects the output command in the CA protocol layer.
3. On-site operators can decide the upper and lower limits of remote control operation
before sending permission commands.

Chapter 5. Operator Intervention System

Figure 5.1: Procedure for implementation of proposed operator intervention system.

59

Chapter 5. Operator Intervention System

5.4

60

Basic Technology for Operator Intervention System

In this section, the underlying systems used by the proposed operator intervention
system are described in detail.

5.4.1

Virtualization Technology

Virtualization technology allows the sharing of hardware resources through cloud computing. Some examples of such virtualization technology are KVM, Xen, Hyper-V, andVMware, and such software is widely used in many situations. Server virtualization isa technology where the OS of several virtual guests can be run in one physical server(see Fig. 5.2). In order to enhance the efficient operation of server hardware resources,
the software listed previously can be used also to construct accelerator control systems[88–93]. On the other hand, such an operator intervention system needs to consist of
several servers with low server load. Therefore, VMware vSphere Hypervisor, whichis free virtualization software produced by VMware Inc., has been adopted consideringsystem security and ease of system construction.

5.4.2

EPICS Process Variable Gateway

In the EPICS-based system, the PV gateway is used for its feature as a proxy system for
the CA connection. In general, the PV gateway is installed for the purpose of relaying
the CA servers, such as EPICS IOC, and CA clients, to reduce the network load in the
EPICS-based system (see Fig. 5.3). In addition, access restriction, namely the EPICS
Access Security Group (ASG), is implemented in the PV gateway as well as the EPICS
IOC. ASG is described in the next section. In the proposed operator intervention system,
the WebSocket server always connects to the EPICS CA server via the PV gateway.

Chapter 5. Operator Intervention System

Figure 5.2: Outline of virtualization technology.

Figure 5.3: Outline of PV gateway.

61

Chapter 5. Operator Intervention System

5.4.3

62

EPICS Access Security GroupEPICS ASG is a security function for implementing restrictions to access the EPICS
IOC and PV gateway through the use of a configuration file [50]. Using ASG, it ispossible to realize access permission and read/write access through a host name and alogin name. An example of the configuration file is as follows.
1

UAG ( uag ) { user1 , user2 }2

HAG ( hag ) { host1 , host2 }3

ASG ( DEFAULT ) {4

RULE (1 , READ )5

RULE (1 , WRITE ) {UAG ( uag )6

HAG ( hag )

7

}

8
9

}

Listing 5.1: Example of the configuration file for ASG.

The previous sample file configures all hosts to be allowed to read PV values. In
addition, the sample file configures host1/host2 as the hostname, user1/user2 as thelogin name, and grants the hostname and login name permission to write PV values. Byutilizing the login name and hostname used by the client PC, ASG can restrict access to
the CA server. However, it is not possible to use ASG as a complete security mechanism
because the login name and hostname depend on system configuration, and malicious
impersonation is possible.

5.5

System Policy

The operator intervention system can control the availability of remote operations bysending system flag values to the MySQL database. The access control mechanism
for the CA protocol is utilized by the access control security system of the PV gateway. In particular, a daemon polls the database and updates the security access file
(GATEWAY.pvlist) with information that corresponds to the flag values dynamically.
A flowchart that illustrates system operation for remote connections is depicted in Fig.
5.4 and described as follows:

1. Remote users send EPICS PV requests to the operator intervention system tocontrol the output, such as caPut command, via WebSocket (Phase flag 0).
2. Then, on-site operators ascertain whether the requested PVs are available forcontrol (Phase flag 1).

Chapter 5. Operator Intervention System63

3. If control is possible, a command to accept is sent by on-site operators (Phase flag2); otherwise, a refuse command is sent (Phase flag 3).
4. On-site operators can terminate remote operation at any time, for example, if
problems with the accelerator arise (Phase flag 3).
5. A process that manages the time duration of the remote operation sends a timeout
command after a certain period has elapsed (Phase flag 4).

Figure 5.4: Flowchart illustrating operation of system for remote connection.

Chapter 5. Operator Intervention System

5.6

64

System Design

The proposed operator intervention system was developed utilizing virtualization technology and the EPICS PV gateway described in Section 5.3.2. In this section, the systemdesign for the operator intervention system is described.

5.6.1

Required Services

The operator intervention system comprises important services that run on several
distributed hosts. A chart of the system is depicted in Fig. 5.5. The MySQL service isnecessary to store the flag value via an internal network. The Apache web server andnetwork file system (NFS) is run on an application server. NFS is used to share log
files among the WebSocket server, the Apache web server, and the PV gateway via theinternal network, as well as the MySQL service. WebSocket server programs developed
using Node.js run on a separate host that connects to the MySQL database and the
EPICS IOCs via the PV gateway.

5.6.2

Preventing Impersonation

In order to make a test installation of multi-layer network structures, virtualization
technology is utilized in order to prevent malicious impersonation. The access control
security system does not comprehensively cover access from the Internet because EPICS
is not able to completely prevent impersonation (see Section 5.3.3). In general, a defensein-depth network system with a multi-layer network structure is one of the most useful
methods for improving network security. In this system, the author implemented amulti-layer network structure that consists of servers running in a virtual environment
on one physical server. In the system, network attacks and unlawful access are thwarted
using a defense-in-depth network to prevent access from a physical network (see Fig.
5.6).
The author implemented a virtual environment using VMware vSphere Hypervisor 5,
which can be used free of cost. The specifications for the physical server are listed inTable 5.1. The author used CentOS 5.9 as the operating system for the virtual machines.
On the other hand, the defense-in-depth network will be composed of multiple physicalservers in the real implementation.

Chapter 5. Operator Intervention System

65

Figure 5.5: System chart showing operator intervention system, EPICS IOCs, and
WebSocket server.

CPU
MemoryStorage
Ethernet

Intel Core i7-3770
16G Byte
SATA
Dual-network connection
Table 5.1: Physical server specifications.

5.6.3

Setting Upper/Lower Limits

In general, EPICS utilizes DRVH/DRVL fields to set the upper and lower limits of theanalog output (ao) records [50]. In this case, however, utilizing the DRVF/DRVL fieldsof the EPICS IOCs to control the system network is not wise because it might cause
problems, such as lack of parameter settings, in normal operation with remote users.
As a result, the author implemented the function by modifying the source code of the
PV gateway. Consequently, the customized PV gateway has a GATEWAY.limit file toChapter 5. Operator Intervention System

66

Figure 5.6: Network chart of system in VMware virtualized environment.

set the upper and lower limits, as well as GATEWAY.pvlist and GATEWAY.access files
for access control.

Chapter 5. Operator Intervention System

5.6.4

67

UIs

The author developed the interface of the operator intervention system as a web application using AJAX because such interface is not required to be as responsive as the
WebSocket-based OPI. The UI of the operator intervention system is shown in Fig. 5.7.
On-site operators can verify the requested EPICS PVs, completed PVs, and in-operation
PVs in one view. Similarly, all the access logs and caPut logs are available for verificationby the on-site operator at any time.

Figure 5.7: User interface for operator intervention system. On-site operator cancontrol remote operation through EPICS PVs with this UI.

5.6.5

Security for WebSocket-side

The author ensures secure WebSocket access to the accelerator network from the WAN
using a VPN and authentication with SSL. In addition, accelerator operators have tocomprehensively verify a user (via login ID) that seeks access to the network and accessroute (e.g., full domain of Internet providers, etc.) before WebSocket control is allowed.

Chapter 5. Operator Intervention System

5.7

68

Similar System

A new wide area remote control system (WARCS) with a concept similar to the proposed
operator intervention system was implemented at SPring-8 in 2012 [94]. At SPring-8, an
old version of WARCS that uses Zebedee, a tunneling tool via VPN, was implemented forremote maintenance via WAN [95]. However, the system has the following limitations.

1. In some cases of network routing, WARCS cannot ensure the network port usedby Zebedee.
2. To use a Zebedee-based VPN for WARCS, a dedicated client terminal is necessary.

To solve these limitations, WARCS was upgraded using SSL-VPN, which is one of
the standard cryptographic technologies. In addition to being used as a terminal for
typical PCs, the main characteristic of WARCS is that the on-site staff responsible forthe operation of the accelerator issue SSL-VPN accounts. Because intervention of onsite staff is necessary for security, such a characteristic of the new WARCS is a concept
similar to the operator intervention system described in this chapter. A comparisonof the features of the new WARCS and the proposed operator intervention system aresummarized in Table 5.2.

Object controlled byon-site staff.
UI
PurposeOperator Intervention System
Control I/Os via WebSocket
Web
Security for remote operation in EPICS-based control system.

New WARCS
VPN accounts
Web
Security for trouble shooting in control system at
SPring-8.

Table 5.2: Comparison of features between new WARCS and proposed operator intervention system.

Chapter 6

System Implementation
The control system for the RIKEN 28 GHz SC-ECRIS consists of a distributed control
system based on EPICS and RIBF. To maintain beam quality for extended beam-service
time at RIBF, beam tuning is required in order to prevent subtle changes in 28 GHz
SC-ECRIS conditions. Once this is achieved, it should be possible to check conditions
and operate the ion source remotely at any time. The author has designed WebSocketbased OPIs to control the ion source remotely; however, for access and control fromoutside the facility, suitable access security, policies, and methods are required. Therefore, the author implemented an operator intervention system that allows access to the
network safely from external sites with permission from on-site accelerator operators in
the control room.

6.1

Target Component for the Implementation

In this section, an ion source, which is a component of a heavy ion accelerator facility
such as RIBF, is described for implementation in this study.

6.1.1

RI-Beam Factory Project

To provide the world’s most intense radioisotope (RI) beams over the entire range ofatomic masses of heavy ions, the RIBF project was started at RIKEN [96, 97]. This
accelerator facility consists of four ring cyclotrons that include a superconducting ring
cyclotron, and two linear accelerators with Azimuthally Varying Field (AVF) cyclotrons
as injectors. In order to increase the energy of the heavy ion beams from the RIKEN
Ring Cyclotron (RRC: K 540 MeV) in the RARF project, three new ring cyclotrons-a
fixed-frequency ring cyclotron (fRC: K 570 MeV), an intermediate stage ring cyclotron
69

Chapter 6. System Implementation

70(IRC: K 980 MeV), and a superconducting ring cyclotron (SRC: K2600 MeV) have been
updated in the RIBF project [98]. The schematic layout of the RIKEN RIBF is shownin Fig. 6.1.
In the RIBF, variable heavy ion beams, from helium to uranium, have been provided
for experiments. However, the capability of the ion source has a decisive influence on
accelerator facility performance, such as beam intensity. Therefore, the ion source is oneof the most essential components that constitute the heavy ion accelerator facility.

Figure 6.1: Schematic layout for RIKEN RIBF. 28 GHz SC-ECRIS was constructedin the injection line for RIBF at RIKEN.

6.1.2

Electron Cyclotron Resonance Ion Source

The kinetic energy per nucleon E of heavy ions accelerated by the cyclotron is represented by the following formula:

E = K(q/M )2(6.1)

where K is an accelerator-specific value that is determined by the strength of the magnetic field and the size of the cyclotron, q is the valence, and M refers to the mass
number. In the above formula, in order to meet the demands of heavy ions with larger
kinetic energy, ion beams for which ’q/M ’ is larger (i.e., highly charged beams) are
required.
Currently, ECRIS is the best equipment that can provide stable direct current highly
charged beams for heavy ion accelerators [99]. The main components required for ECRIS
control are as follows.

Chapter 6. System Implementation

71

1. Material supply (gas, rod, oven, etc)
2. Ionization (plasma production)
3. Extraction
Heavy-ions are generated by introducing material supply in plasma using electron
cyclotron resonance, and the heavy-ions are extracted using an electric field (See Fig.
6.2). In RIBF, EPICS is used for these controls; the number of control points for 28
GHz SC-ECRIS is approximately 1,000 points EPICS records. In the next section, the28 GHz SC-ECRIS, which is one of the ECRIS used in RIBF, is described.

Figure 6.2: Outline of the ECRIS.

6.1.3

28GHz SC-ECRISThe performance of the ion source is an important factor in determining the overall
performance of a heavy ion accelerator. At the RIKEN RIBF, the 28 GHz SC-ECRIS(see Fig. 6.3, 6.4) was constructed in 2009 in an effort to increase the uranium-ion beam
intensity [100, 101]. In reality, the stability of the beam is important in terms of beamservice availability for experiments at RIBF, and periodic fine-tuning of parameters suchas the RF power and gas pressure is required to ensure a high-intensity beam with longterm stability. Therefore, it is common for ion source developers to telephone on-siteaccelerator operators to check the status of the 28 GHz-SC-ECRIS when not present
at the facility. However, ion source developers could obtain more detailed information
regarding the status of the source, relay operational instructions, and maintain beam
quality if it were possible to access the control system from outside the facility.

Chapter 6. System Implementation72

In addition, because the ion source includes all components (e.g., RF, vacuum, magnet,
and beam diagnostics) that compose the accelerator, it is suitable as a target for test
implementation of the operator intervention system and WebSocket-based OPIs via a
WAN link. Thereby, the author implemented WebSocket-based OPIs and an operator
intervention system for the 28 GHz SC-ECRIS control system.

Figure 6.3: Photograph of the RIKEN 28GHz SC-ECRIS.

6.2

Implementation BackgroundAs described in Chapter 4, traditional Web applications that use AJAX lack the interactivity required to act as OPI for some accelerator hardware. However, the WebSocket
protocol can be implemented to improve interactive performance. This protocol achieves
bidirectional communication between a Web server and a browser, unlike the periodic
polling access of AJAX. Bidirectional communication can be utilized to develop an OPI
that can both monitor and control ion-source devices. Based on the description in
Section 4.6, the author has successfully developed a WebSocket-based client system tomonitor the real-time performance of the EPICS-based system via a Web browser.

Chapter 6. System Implementation

73

Figure 6.4: Section View of RIKEN 28GHz SC-ECRIS.

There are clear advantages to the developed client system; however, there are still someproblems that need to be addressed, namely, hazards associated with access via WebSocket and operator errors. Focusing on the latter, operator errors can arise because it
is difficult to provide a complete picture of the system status verbally over the telephone(seeing RIBF parameters on control-room PC displays is of much greater benefit) andbecause operating the 28 GHz SC-ECRIS without real-time response is difficult, even if
remote operators can access snapshots of the GUI screens as image files. To prevent operator errors from occurring, accelerator conditions that include the 28 GHz SC-ECRIS
need to be understood in their entirety. Because accelerator operators in the control
room have the best understanding of the RIBF accelerator conditions, their judgment is
indispensable to ensure that devices are controlled safely. On-site accelerator operators
must then be allowed to intervene in remote operation. The WebSocket-based clientsystem is designed such that output control by the remote ion source developer always
requires the permission of an on-site accelerator operator who can intervene at any time.

Chapter 6. System Implementation

6.3

74

28GHz SC-ECRIS Control System

In this section, the control system of the 28 GHz SC-ECRIS in normal operation is
described for comparison with remote operation.

6.3.1

Outline of Control System for 28 GHz SC-ECRIS

Because the 28 GHz SC-ECRIS control system is based on EPICS as part of the RIBF
control system, the same network system is used as that for the RIBF control system. To
control the main devices, such as gas valves and rod position, in the 28 GHz SC-ECRIS,
a combination of a Yokogawa F3RP61-2L module that acts as an EPICS IOC and FA-M3
PLCs (Yokogawa Electric Corp.) are adopted with the embedded EPICS technology (seeSection 2.3.3). A vacuum control system and a beam diagnostic system are constructedby EPICS using Linux-based IOCs connected with N-DIMs via Ethernet [102]. The
bending magnet control system comprises a VME-based IOC that communicates withNIO [33]. The outline of the 28 GHz SC-ECRIS control system is shown in Fig. 6.5.

Figure 6.5: Outline of the 28GHz SC-ECRIS control system. Hardware dependent
protocols are unified by CA protocol.

6.3.2

Client System

To facilitate information exchange among different types of controllers, the protocol forcontrollers such as FA-M3, N-DIMs, and NIOs is unified through a CA protocol fromthe client’s perspective. A constructed x-y plotter chart that uses EDM as described in
Section 2.4.3.1 is shown as a client in Fig. 6.6. Because there are numerous parameters
related to the operation of the 28 GHz SC-ECRIS, maintaining records in a paper
notebook is not feasible. To save the 28 GHz SC-ECRIS parameters in an effective
manner, a paperless log system that uses wiki (see Fig. 6.7) is adopted. Wiki is aChapter 6. System Implementation75

system that can edit and create hypertext documents on a Web server through a Web
browser; knowledge of HTML is not required in this case.
Using MyDAQ2 described in Section 3.2.3 as the DAQ system for the 28 GHz SCECRIS, EPICS values such as vacuum, extraction current, and RF power are recordedin the MySQL database every 10 s. Because graphics and text can be easily displayed in
Web browsers, the Web applications have a user-friendly GUI and a Wiki-based system.
DAQ based on MyDAQ2 can be used to reduce the turnaround time during systemconstruction.
To manage differences between terminals used in the accelerator control room andremote locations, a cross-platform software environment should be adopted (for example,
less common software such as the X Window System might not necessarily be installed
on a remote computer, as it is on the control-room Linux PCs). In this regard, Webbased technology has the advantage that any device with a browser, e.g., a PC, mobile
phone, tablet, or TV, can be used, and Web-based technology has been shown to be
useful for conveying status information from the 28 GHz SC-ECRIS control system.

Figure 6.6: An x-y plotter chart application constructed by EDM.

6.3.3

Network System

A closed network is more reliable for accelerator control from the perspective of information security. The RIBF control system is also constructed on a closed network that

Chapter 6. System Implementation

76

Figure 6.7: Wiki page with a screen capture from the 28GHz SC-ECRIS main control
GUI panel.

is completely disconnected from all other networks, including laboratory Intranets used
for daily research activities [103]. In this network system, it is difficult for staff to obtainparametric data, such as from the 28 GHz SC-ECRIS, from their offices because accessports to the network for EPICS are not installed in the member’s offices. However, fromthe perspective of system management and ease of system construction, Web communication is suitable for providing information as mentioned in Section 6.2.2. To meet thisrequirement, a combined system with reverse proxy servers for Web communication anda firewall was constructed for providing accelerator information to the office network
while ensuring secured access. Installation of reverse proxy servers in front of real Web
servers is a widely used prescription for security and caching.
In order to implement the system, reverse proxy servers and firewalls were constructed.
The outline of the system is shown in Fig. 6.8. The primary feature of the system isthat access from the office network are masked from the Web servers used for Wiki and
MyDAQ2 behind the reverse proxy servers.
Because it is possible to analyze data such as vacuum and gas pressure efficiently using
Web services from offices as well as the control room, the system has allowed more

Chapter 6. System Implementation77

effective use of working time for 28 GHz SC-ECRIS operation [104]. An outline of theroutine work in 28 GHz SC-ECRIS operation and the relationship with Web services
are shown in Fig. 6.9.

Figure 6.8: Network diagram of Web communication via a reverse proxy server between office network and control system network.

Figure 6.9: After operation of the 28GHz SC-ECRIS in the control room, the data
are downloaded to office room via Web services for analysis and R&D.

Chapter 6. System Implementation

6.4

78

WebSocket-based client system for 28GHz SC-ECRISControl System

As described in Chapter 4, WebSocket enables interactive response using bidirectional
communication, thus eliminating the disadvantage of traditional HTML. Web browsers
such as Firefox, Chrome, and Internet Explorer can be used as a cross-platform environment with the implementation of a WebSocket-based OPI. This allows the user toaccess OPI not only from a PC-based Web browser, but also from other devices, suchas Android and Apple tablets and mobile phones. After satisfying the conditions forinteractive access to the EPICS-based system, the author implemented the WebSocket
server that connects to the EPICS IOC via CA protocol as a Web-based OPI for 28 GHz
SC-ECRIS operation. Figure 6.10 shows a chart of the whole system. The WebSocketbased client system is comprised of an HTTP server, HTML and JavaScript files, and a
WebSocket server. The WebSocket server obtains the parameters as PVs from EPICS
IOC by calling the CA API.

Figure 6.10: System diagram of WebSocket-based OPI in EPICS-based control system.

Chapter 6. System Implementation

6.5

79

Implementation Results

As described in Chapter 5, the operator intervention system consists mainly of a PV
gateway provided by EPICS collaboration, a database, and Web applications. An outlineof the implemented system is as follows. First, on-site accelerator operators grant permission to each PV output that corresponds to the hardware of the 28 GHz SC-ECRIS
control system. If PVs have not been granted permission, the operator intervention
system always rejects the output command at the CA protocol layer. GUI for the
WebSocket-based OPI is shown in Fig. 6.11. Operation requests to on-site operators
are sent by clicking the buttons that correspond to the EPICS PVs. In addition, on-siteaccelerator operators can check access logs for all remote operations with output devices
and monitor numerical values for the EPICS PVs.
Remote operation of the 28 GHz SC-ECRIS control system was tested using a WAN
emulator with remote users and on-site operators in the same control room, but accessing
different network environments [105] (see Fig. 6.12). The WAN emulator creates anothernetwork with variable performance. As a result, the ion source developer was successfully
able to operate part of the 28 GHz SC-ECRIS (gas valves, RF power, position of anelectrode, and so on) remotely without serious problems.
Using this system, the author confirmed that output instructions did not reach the
EPICS IOC without permission from the on-site operator. In addition, remote operation
was performed satisfactorily with a network latency of 200 ms, which is approximately
the same network latency between Japan and the USA. Therefore, networking performance tolerance should allow this system to be used over networks provided by almost
all Japanese Internet service providers [106].
The author has to note that “debugging” HTML and JavaScript programs proves to be
more important for preventing operational errors than other general Web applications.
In the case of EPICS native OPI coded in the C language, the compiler should verify
incomplete syntax of source codes. However, even when the HTML or JavaScript code
has a “bug”, the code can be run in a Web browser in some situations.

Chapter 6. System Implementation

Figure 6.11: User interface of WebSocket-based OPI. By clicking on buttons, operation requests are sent to the on-site operators.

80Chapter 6. System Implementation

Figure 6.12: Test environment with another network-emulated WAN. Gray area indicates implemented system.

81

Chapter 7

Discussion
7.1

Performance Measurement of WebSocket-based OPI

For comparison of performance in remote operation, network bandwidth was measured
for WebSocket-based OPI and AJAX-based OPI. In order to measure the network bandwidth, OPIs for the 28 GHz SC-ECRIS operation that is described in Chapter 6 wereused. A feature of the WebSocket server developed with the Socket.IO library is thatthe system behaves as AJAX access when a WebSocket-unsupported browser connects
to the WebSocket server. Utilizing the feature of Socket.IO, the difference in network
bandwidth obtained from accessing the system in two different ways, AJAX and WebSocket, were compared with the same OPI. In OPI, 30 numerical values, such as gas
valves positions, rod position, and extraction power supply, were monitored at intervalsof 0.1 s. Then, four numerical values for vacuum were monitored at intervals of 4 s. In
this paper, an example of a response through WebSocket and AJAX access is described
below.
1

5::/28 gecris / index :{ " name " : " ca " ," args " :[{ " gref_pwr_v " : " -0.0325 " }]}Listing 7.1: Example of responce data in the WebSocket server.

The same data frame is received by the Web browser regardless of the difference inaccess methods. First, the author measured latency in the case of AJAX access to OPI
using Wireshark [107], which is a network packet analyzer software. In the case of UI for
the remote operation of 28 GHz SC-ECRIS, the result of the AJAX-polling measurement
through LAN is a latency of 0.5 s that was observed on average upon one access.
In order to test in a network compatible with the network used for implementation of
the operator intervention system described in Chapter 5, a WAN emulator was used asa measuring instrument.
82

Chapter 7. Discussion

7.1.1

83

Network Bandwidth Measurement for WebSocket

In this study, to measure network bandwidth, a WAN emulator (Linktropy Mini2)
produced by Apposite Technologies, Inc. [108] was used. Across the WAN emulator,
LAN A shows the network of the virtual server side; LAN B shows the office network(see Fig. 7.1). The author measured the bandwidth of WebSocket-based OPIs throughLAN using Google Chrome version 34 as the Web browser. The graph in 7.2 shows the
network bandwidth when a single client is connected to the WebSocket server to monitor
accelerator parameters.
The graph in Fig. 7.3 shows that the bandwidth requires 84.39 kbps with an average of
10 min from LAN A to LAN B in the case of a single client. On the other hand, network
bandwidth to LAN B from LAN A requires 22.42 kbps only with an average of 10 min.

Figure 7.1: Network chart for measuring network bandwidth.

Chapter 7. Discussion

Figure 7.2: Graph of network traffic for WebSocket-based OPI.

84

Chapter 7. Discussion

7.1.2

85

Network Bandwidth Measurement for AJAX

In the case of AJAX-based OPI, the network bandwidth was measured in terms comparable to the WebSocket-based OPI described in Section 6.5. Mozilla Firefox version
3.6.13 for Linux was adopted as the Web browser. The graph in Fig. 7.3 shows the network bandwidth when a single client is connected to the WebSocket server with AJAX
to monitor accelerator parameters.
As shown in the graph, the bandwidth requires 42.0 kbps with an average of 10 min
from LAN A to LAN B in the case of AJAX access from a single client. On the other
hand, network bandwidth to LAN B from LAN A requires 83.51 kbps with an averageof 10 min.

Figure 7.3: Graph of network traffic for AJAX-based OPI.

Chapter 7. Discussion

7.1.3

86

Network Bandwidth Measurement for X Window System

As described in Section 1.3.4, the X Window System is widely used for OPIs by exporting
GUIs to a remote computer in the case of Web technology. In reality, the main GUI
panel for the 28 GHz SC-ECRIS control is constructed with EDM provided by EPICS
collaboration (see Fig. 2.13). Because the specifications of the protocol are completely
different from each other, it is difficult to compare them directly. However, to confirm theadvantage of WebSocket-based OPIs, network bandwidth for EDM-based OPIs, which
are implemented for the 28 GHz SC-ECRIS daily operation, were measured. For the
X server, Cygwin-X that runs on Microsoft Windows was adopted, and the X client(used by EDM installed in a virtualization server) is based on CentOS 5.9. This EDMbased OPI has approximately the same number of monitored parameters compared with
WebSocket-based OPI for 28 GHz SC-ECRIS.
The graph in Fig. 7.4 shows the network bandwidth in the case of a single EDM-based
OPI. This graph shows that more than approximately 140 kbps is necessary as network
bandwidth to display the 28 GHz SC-ECRIS main GUI panel via the X Window System.

Figure 7.4: Graph of network traffic for OPI constructed with EDM based on X
Window System.

Chapter 7. Discussion

7.1.4

87

Measurement Results

The measurement results from requests to the WebSocket server show that AJAX-based
OPI requires eight times more network bandwidth compared to WebSocket-based OPI.
However, AJAX-based OPI can receive only half the data volume that WebSocket-based
OPI receives. On the other hand, almost all the values to be monitored by EPICS IOC
are changed at intervals of 0.1 s (SCAN 0.1 second), and one AJAX communication
requires approximately 0.5 s to receive data in LAN after transmitting data requests.
Therefore, this means that a 0.5 s delay occurs regardless of LAN-links, and AJAX-based
OPI can receive only half of the data volume because timing of the AJAX transmission
is too slow compared with timing of the SCAN in EPICS IOCs.
Moreover, if this AJAX-based OPI is used via WAN-links, it is clear that the delay
increases. For this reason, for AJAX-based systems, although a large amount of network
bandwidth is necessary for transmitting requests, a few amount of data is received.
Thereby, AJAX-based OPI is an inefficient method compared to WebSocket-based OPI.
On the other hand, EDM-based network traffic shows that the network bandwidth fordata requests requires only approximately 13 kbps; however, a relatively large 140 kbps
is necessary to receive data. This is because an event for the X Window System is
not generated if the operator does not move a window or the mouse cursor, or clicks
the mouse. On the other hand, the X client sends packets to the X server to request
drawables. The network bandwidth for sending packets (from LAN A to LAN B) is
increased by the increase of CA events because requests for drawables are sent to the X
server by changing the record value on EPICS IOC.
As a result, AJAX-based OPI is not suitable for fast responses, such as “field(SCAN,“.1
second”),” in EPICS Runtime DB, and it is difficult to use EDM-based OPI using the X
Window System from WAN-links because data received for the X server is a large frame.
Thereby, the author considers that WebSocket-based OPI should be implemented for themost efficient remote operation.

Chapter 7. Discussion

7.2

88

Future Improvements of WebSocket-based OPI

As described in Section 4.5, the author developed Node-CA, which is interfaced between
CA and Node.js by calling CA API. However, the current version Node-CA implements
caMonitor, caGet, and caPut, which are the main functions of CA API. In particular,functions related to arrays, such as the caGet array or the caPut array, were not implemented in this study because these functions have no need for 28 GHz SC-ECRIS
control. In the future, it is necessary to update Node-CA for full implementation ofother EPICS-based applications coded in Node.js.
Moreover, existing GUI managers, such as CSS/BOY and MEDM, have strong points
in EPICS-based control systems, such as software productivity, because there is no needto write code such as drawing software; therefore, it is possible to construct GUI applications for EPICS-based control systems. On the other hand, compared with the development of X applications and Java Swing-based applications, there is a small amount,
but WebSocket-based OPI is necessary for coding in HTML in the case of this study. In
order to broaden the base of users who employ the WebSocket-based OPI, the author
would like to develop a display manager that supports WebSocket-based OPI.
The software that is a result of this study is expected to be released as open source
software in the near future.

7.3

Challenges for the future

For remote operation of accelerators, the author confirmed the effectiveness of WebSocketbased OPI in Section 6.4. On the other hand, data archive systems are widely used fordaily accelerator operation in local control rooms. As described in Section 3.2.5, Webbased systems are used for data archive viewers. In order to determine the cause ofaccelerator conditions by comparing different types of archived data, data archive viewers are one of the frequently used applications in accelerator control systems.
However, there is an issue for using data archive viewers via remote operation because
the amount of data transfer increases if the search scope is spread. For example, aconsiderable amount of time might be required to search and display a year’s worthof vacuum data via WAN-links. In this case, the author considers that using machinelearning is an effective method for searching and transferring data via WAN-links.
In many cases, operators utilize data archivers to determine the time when datachanged and the amount by which such data changed, for example, vacuum values.

Chapter 7. Discussion

89

Thus, if operators can determine the time when such data changed, it is possible toreduce the amount of data transferred.
For this purpose, utilizing the framework of machine learning that has a function for
anomaly detection, such as Jubatus [109], will be a key point for constructing an effective
system.

Chapter 8

Conclusion
From the need for operation and maintenance, some accelerator research facilities havestudied wide area environments for remote operation since 2000. Such facilities adopted amechanism that requires considerable system resources, for example, dedicated networks
and special-purpose terminals, for this remote operation. The purpose of this study isto perform the implementation of remote operation in control systems using EPICS as
middleware, even in environments with limited resources.
On the other hand, Web-based systems have multi-purpose potential in local area networks; therefore, some projects have attempted to develop OPI using traditional Web
technology, although their main purpose is to monitor accelerator parameters throughstatic access. However, Web-based OPI that uses traditional technology, such as AJAX,
is not always available for use in wide area environments because previous systems havedisadvantages in interactive performance. Thereby, the author proposed a hypothesisthat Web-based systems might be strong candidate methods for remote operation if theshortcomings of traditional Web-based OPIs can compensated through this study. For
this reason, WebSocket-based OPIs for EPICS-based control systems and an operator
intervention system, which is a new approach compared with previous works, was developed in this study; moreover, the proposed WebSocket-based OPIs increase software
portability and reusability dramatically without damaging the advantage of Web-based
systems.
In addition, local control rooms have aggregated information; therefore, the judgmentof on-site operators is extremely important from the perspective of safety systems. To
solve this issue in remote operation via WebSocket, the author discussed the development and implementation of an operator intervention system for remote operation using
WebSocket-based OPIs. This intervention system ensures that operational errors areminimized and that the accelerator is operated safely during remote operation. Because
90Chapter 8. Conclusion

91

the system was implemented for RIKEN 28 GHz SC-ECRIS remote operation, the author confirmed that this new operator intervention system via WebSocket-based OPIs
is a useful method for ensuring the physical security of operational errors.
On the other hand, remote participation according to a multi triangle-model and the
GAN operational model, which is an extensive remote operation of the facility, were discussed for the GAN project. The GAN operational model means a relationship between
accelerator control rooms, accelerator technical infrastructure, and home laboratories forcollaboration; such a relationship is required for the effective direction of significantly
large physics projects, such as ILC.

• In virtual accelerator control rooms, multiple operators from the operation teamare deployed.

• For repair and maintenance, the accelerator technical infrastructure located nearto the accelerator is used by a small core team.

• Experienced staff (or teams) work in technical monitoring, fault analysis, R&D,andplanning of repair and maintenance in home laboratories.

Figure 8.1: Remote operation model using WebSocket and operator intervention
system in this study.

One of the continued efforts of the GAN project, efficient communication protocols
such as remote method invocation and messaging services, was discussed. On the other
hand, WebSocket-based OPIs and the operator intervention system were successfully
implemented for the remote operation of 28 GHz SC-ECRIS control system in thisChapter 8. Conclusion

92

study. As shown in Fig. 8.1, this system consists of a counterpart system based on the
GAN operational model, although the system scale is much different between the GAN
model and the 28 GHz SC-ECRIS remote operation.
Therefore, the proposed method, which combines HTTP and WebSocket, is a primary
candidate as the communication layer used on the Internet for remote operation. Becauseof advantages in traditional implementation and new implementation with WebSocket,
remote operation needs to be realized through a suitable approach that uses both the
HTTP and WebSocket protocols. For example, for implementation ease, traditional Web
applications should be adapted if a fast response is not required for remote operation.
For these reasons, a combination of HTTP and WebSocket, and the on-site operator
intervention are essential methods for remote operation.
Finally, the author hopes that this system concept and system design become partof the absolutely necessary features of standard models for control systems in futureaccelerator projects.

Appendix A

Achievement
A.1

International Conference• Akito UCHIYAMA, Kazuro Furukawa, and Yoshihide Higurashi, ”Operator Intervention System for Remote Accelerator Diagnostics and Support” Proc. of
ICALEPCS2013, San Francisco, CA, USA, Oct. 2013, pp.832-835, TUPPC110.
(Poster)• Akito Uchiyama, Kazuro Furukawa, Yoshihide Higurashi, and Takahide Nakagawa,”Implementation of an Operator Intervention System for Remote Control of the
RIKEN 28GHz Superconducting ECR Ion Source”, Proc. of ICIS2013, Chiba,
Japan, Sep. 2013, (Poster) (To be published in Review of Scientific Instruments)• Akito Uchiyama, Kazuro Furukawa, and Yoshihide Higurashi, ”EPICS Channel
Access Using WebSocket”, Proc. of PCaPAC2012, Kolkata, India, Dec. 2012,
pp.7-9, (Oral)• Akito Uchiyama, Kazuro Furukawa, Yoshihide Higurashi, Kazutaka Ozeki, Misaki
Komiyama, and Takahide Nakagawa, ”Design of Web-based Interface to RIKEN 28
GHz Super-conducting ECR Ion Source and the Future Plan”, Proc. of ECRIS2012,
Sydney, Australia, Sep. 2012, (Poster)A.2

Domestic Conference• Akito Uchiyama, Kazuro Furukawa, Yoshihide Higurashi, Takahide Nakagawa,”Development of Operator Intervening System for remote Accelerator diagnostics
and support”, Proc. of Particle Accelerator Society of Japan 2013, Nagoya, Japan,
Aug. 2013, (Poster)93

Appendix A. Achievements

94• Akito Uchiyama, Kazuro Furukawa, Yoshihide Higurashi, ”Development and Implementation of EPICS Channel Access Client with Real-time Web using WebSocket”, Proc. of Particle Accelerator Society of Japan 2012, Toyonaka, Japan,
Aug. 2012, pp.218-221, (Oral)

A.3

Award

• Oral Presentation Award (Particle Accelerator Society of Japan, 2013).

Appendix B

Source Code
B.1

Channel Access for Node.js

B.1.1

nodeca.cc1//======================================================================//

2//2012 -04 -03 [ A . UCHIYAMA ]3//

File : nodeca . cc ver .0.3.2

4//

The Graduate University for Advanced Studies ( KEK )5//mailto : akito12@gmail . com

6//

Required node . js version is v0 .6.11 or later .

7//======================================================================//

8# include < queue >

//
//

9
10

// node headers

11# include < v8 .h >12# include < node .h >13

# include <uv - private / ev .h >14# include < pthread .h >15# include < unistd .h >1617# include < stddef .h >18

# include < stdlib .h >19

# include < stdio .h >20

# include < string .h >21

# include < cstring >22
23// EPICS

24# include " cadef . h "25# define M A X _ B U F _ L E N 512

26
27

using namespace node ;28

using namespace v8 ;

29
30
31

//

//

// handles required for callback messages

95//Appendix B. Source Code96

32

static pthread_t thread_id ;33

static ev_async e i o _ t c a m o n i t o r _ n o t i f i e r ;34
35

Persistent < String > callback_symbol ;36

Persistent < Object > module_handle ;37
38

std :: queue < char * > msg_queue = std :: queue < char * >() ;39

pthread_mutex_t queue_mutex = P T H R E A D _ M U T E X _ I N I T I A L I Z E R ;40
41

struct thread_data {const char * _time ;42
43

const char * _pv ;44chid _channelId ;

45

};

46
47

const char * ToCString ( v8 :: String :: Utf8Value & value ) {return * value ? * value : " < string conversion failed > ";

48
49

}

50
51

// - - - - - - - - - - - - - - - - - - - - - - - -//

52

//

53

// - - - - - - - - - - - - - - - - - - - - - - - -//caGet method//

54

Handle < Value > caGet ( const Arguments & args ) {55

HandleScope scope ;56

const char * usage = " usage : caget ( record_name ) ";57

if ( args . Length () != 1 || ! args [0] - > IsString () ) {58

printf ( " node - ca : % s \ n " , usage ) ;59

return ThrowException ( Exception :: Error ( String :: New (" Node - ca : Bad

60

argument . " ) ) ) ;
61

}

62

v8 :: String :: Utf8Value str ( args [0]) ;63
64// EPICS Channel Access using C lang65

double dData ;66

int iData ;67

char sData [128];68

double * adData ;69chid node_chid ;70

int status , result ;71

SEVCHK ( ca_ con tex t_c reat e ( c a _ d i s a b l e _ p r e e m p t i v e _ c a l l b a c k ) ," c a _ c o n t e x t _ c re at e

72

SEVCHK ( ca_ cre ate _ch anne l (* str , NULL , NULL , 10 , & node_chid ) ," ca_ c re at e _c h ann el73

status = ca_pend_io (5.0) ;74

if ( status != ECA_NORMAL ) {failure " ) ;failure " ) ;return ThrowException ( Exception :: Error ( String :: New (" Node - ca : Channel connect

75

timed out . " ) ) ) ;
76

}

77
78

switch ( ca_field_type ( node_chid ) ) {79

case DBR_ENUM :80case DBR_CHAR :81case DBR_STRING :82

result = ca_get ( DBR_STRING , node_chid , ( void *) & sData ) ;Appendix B. Source Code

97

83

status = ca_pend_io (.5) ;84

if ( status != ECA_NORMAL ) {return ThrowException ( Exception :: Error ( String :: New (" Node - ca : Channel

85

connect timed out . " ) ) ) ;
86

}

87

return String :: New ( sData ) ;break ;

88
89

case DBR_DOUBLE :90case DBR_FLOAT :91
92

result = ca_get ( DBR_DOUBLE , node_chid ,( void *) & dData ) ;93

if ( result != ECA_NORMAL ) {return ThrowException ( Exception :: Error ( String :: New (" Node - ca : ca_get Failure

94.")));}

95
96

status = ca_pend_io (.5) ;97

if ( status != ECA_NORMAL ) {free ( adData ) ;98

return ThrowException ( Exception :: Error ( String :: New (" Node - ca : Channel

99

connect timed out . " ) ) ) ;}100

return Number :: New (( double ) dData ) ;101

break ;

102
103
104

case DBR_SHORT :105

case DBR_LONG :SEVCHK ( ca_get ( DBR_LONG , node_chid , ( void *) & iData ) ," ca_get failure " ) ;

106
107

status = ca_pend_io (.5) ;108

if ( status != ECA_NORMAL ) {return ThrowException ( Exception :: Error ( String :: New (" Node - ca : Channel

109

connect timed out . " ) ) ) ;
110

}

111

return Number :: New (( long ) iData ) ;break ;

112
113

case TYPENOTCONN :114

return ThrowException ( Exception :: Error ( String :: New (" Node - ca : Channel is not115

connecting ( TYPENOTCONN ) . ") ) ) ;break ;

116
117
118

default :119

return ThrowException ( Exception :: Error ( String :: New (" Node - ca : undefined type . "
)));
}

120

return String :: New ( " OK " ) ;

121
122

}

123
124

// - - - - - - - - - - - - - - - - - - - - - - -//

125

//

126

// - - - - - - - - - - - - - - - - - - - - - - -//caPut method

//

127

Handle < Value > caPut ( const Arguments & args ) {128

HandleScope scope ;129

const char * usage = " usage : caput ( record_name , val ) ";130

if ( args . Length () != 2 || ! args [0] - > IsString () ) {131

Appendix B. Source Code98

printf ( " node - ca : % s \ n " , usage ) ;132

return ThrowExceptio n ( Exception :: Error ( String :: New (" Node - ca : Bad

133

argument . " ) ) ) ;
}

134
135
136

v8 :: String :: Utf8Value str1 ( args [0]) ;137

v8 :: String :: Utf8Value str2 ( args [1]) ;

138
139

char

* wdata = * str2 ;

140

chid

node_chid ;141

int status ;int success_val ;

142
143
144

SEVCHK ( ca_ con tex t_c reat e ( c a _ d i s a b l e _ p r e e m p t i v e _ c a l l b a c k ) ," ca_ c on te x t_ cr ea t e

145

SEVCHK ( ca_ cre ate _ch anne l (* str1 , NULL , NULL ,10 ,& node_chid ) ," ca_c r ea t e_ c ha nn el146

status = ca_pend_io (0.5) ;failure " ) ;failure " ) ;147

if ( status != ECA_NORMAL ) {148

return ThrowException ( Exception :: Error ( String :: New (" Node - ca : Channel connect

149

timed out . " ) ) ) ;
}

150
151
152

success_val = ca_put ( DBR_STRING , node_chid ,( void *) wdata ) ;153

if ( success_val != 1) {c a_ c on te x t_ de s tr o y () ;154

return ThrowException ( Exception :: Error ( String :: New (" Node - ca : ca_put failure . "

155

)));
}

156

status = ca_pend_io (0.5) ;157

if ( status != ECA_NORMAL ) {158

return ThrowException ( Exception :: Error ( String :: New (" Node - ca : Channel connect

159

timed out . " ) ) ) ;
}

160
161

c a_ c on te x t_ d e s tr o y () ;162

return String :: New ( " OK " ) ;

163

}

164
165

// - - - - - - - - - - - - - - - - - - - - - - - - - - -//

166

//

167

// - - - - - - - - - - - - - - - - - - - - - - - - - - -//camonitor method

//

168
169
170

void monitorFunc ( struct ev e n t _ ha n d l e r _ a r g s args ) {chid node_chid = args . chid ;171char * pname = strdup ( ca_name ( node_chid ) ) ;172char data [ MAX_BUF_LEN ];

173
174
175

if ( args . status != ECA_NORMAL ) {printf ( " Node : Channel connect timed out \ ’% s \ ’ not found .\ n " , ca_name (node_chid ) ) ;

176
177
178
179

} else {char * pdata = ( char *) args . dbr ;sprintf ( data , " %s ,% s " , pname , pdata ) ;p th re a d_ m ut ex _ lo c k (& queue_mutex ) ;Appendix B. Source Codemsg_queue . push ( data ) ;

180
181

p t h r e a d _ m u t e x _ u n l o c k (& queue_mutex ) ;182

ev_async_send ( EV_DEFAULT_UC_ & e i o _ t c a m o n i t o r _ n o t i f i e r ) ;
}

183
184

}

185
186

// The background thread

187

static void * camonThread ( void * p ) {chid channelId ;

188
189

struct thread_data * _pdata = ( struct thread_data *) p ;190

const char * pv = _pdata - > _pv ;191

double timeout = atof ( strdup ( _pdata - > _time ) ) ;192channelId = _pdata - > _channelId ;193

ca _co nte xt_c rea te ( c a _ d i s a b l e _ p r e e m p t i v e _ c a l l b a c k ) ;

194
195

ca _cr eat e_ch ann el ( pv , NULL , NULL , 10 , & channelId ) ;196

ca_pend_io (.05) ;197

ca_add_event ( DBR_STRING , channelId , monitorFunc , NULL , NULL ) ;198

ca_pend_event ( timeout ) ;199

free ( _pdata ) ; // free malloc

200// thread detach201

pthread_detach ( pthread_self () ) ;202

printf ( " Monitor record is finished .\ n ") ;203

pthread_exit (0) ;204

return NULL ;

205

}

206
207

// callback that runs the javascript in main thread

208

static void Callback ( EV_P_ ev_async * watcher , int revents )

209

{

210

HandleScope scope ;211

assert ( watcher == & e i o _ t c a m o n i t o r _ n o t i f i e r ) ;212

assert ( revents == EV_ASYNC ) ;

213
214

Local < Value > callback_v = module_handle - > Get ( c al l ba ck _s y mb ol ) ;215

if (! callback_v - > IsFunction () ) {216

printf ( " Node - ca : Error not set Callback function \ n ") ;return ;

217
218

}

219

Local < Function > callback = Local < Function >:: Cast ( callback_v ) ;

220
221

// dequeue callback message

222

p th re a d_ m ut ex _ lo c k (& queue_mutex ) ;223

char * number = msg_queue . front () ;224

msg_queue . pop () ;225

p t h r e a d _ m u t e x _ u n l o c k (& queue_mutex ) ;

226
227

TryCatch try_catch ;

228
229

// prepare arguments for the callback

230

Local < Value > argv [1];231

argv [0] = Local < Value >:: New ( String :: New ( number ) ) ;

232
233

// call the callback and handle possible exception

234

callback - > Call ( module_handle , 1 , argv ) ;99

Appendix B. Source Code235

if ( try_catch . HasCaught () ) {236

FatalException ( try_catch ) ;

237

}

238
239

}

240
241
242

// Start the background thread243

Handle < Value > start_camon ( const Arguments & args )

244

{

245

HandleScope scope ;246

struct thread_data * d ;247

if (( d = ( thread_data *) malloc ( sizeof (* d ) ) ) == NULL ) {return v8 :: ThrowException ( v8 :: String :: New ( " Node - ca : Malloc thread error " ) ) ;

248

}

249
250

const char * usage = " usage : camonitor ( record_name , val ) ";251

v8 :: String :: Utf8Value str ( args [0]) ;252

v8 :: String :: Utf8Value cNum ( args [1]) ;253

if (* str == NULL ) {

254
255

printf ( " node - ca : % s \ n " , usage ) ;256

return v8 :: ThrowException ( v8 :: String :: New ( " Node - ca : Bad parameters " ) ) ;
}

257
258
259

// EPICS record name , timeout

260

const char * pv = ToCString ( str ) ;261

const char * time_out = ToCString ( cNum ) ;262

d - > _time = strdup ( time_out ) ;263

d - > _pv = strdup ( pv ) ;264

ev_async_init (& eio_tcamonitor_notifier , Callback ) ;265

ev_async_start ( EV_DEFAULT_UC_ & e i o _ t c a m o n i t o r _ n o t i f i e r ) ;266

ev_unref ( EV_DEFAULT_UC ) ;267// make thread268

if ( pthread_create (& thread_id , NULL , camonThread ,( void *) d ) != 0) {perror ( " pthread_create " ) ;

269

}

270
271

return v8 :: Undefined () ;

272
273

}

274
275

void Initialize ( Handle < Object > target )

276

{

277

HandleScope scope ;278

NODE_SET_METHOD ( target , " camonitor " , start_camon ) ;279

callback_symbol = NODE_PSYMBOL (" callback " ) ;280// store handle for callback context

281

module_handle = Persistent < Object >:: New ( target ) ;282

NODE_SET_METHOD ( target , " caget " , caGet ) ;NODE_SET_METHOD ( target , " caput " , caPut ) ;

283
284

}

285
286
287

extern " C " {static void Init ( Handle < Object > target ) {Initialize ( target ) ;

288
289

}

100

Appendix B. Source CodeNODE_MODULE ( nodeca , Init ) ;

290
291

101

}

Listing B.1: Channel Access for Node.js.

B.1.2

wscript1

srcdir = ’. ’

2

blddir = ’ build ’3

VERSION = ’ 0.3.5 ’

45
6

def set_options ( opt ) :opt . tool_options ( ’ compiler_cxx ’)

78

def configure ( conf ) :9

conf . check_tool ( ’ compiler_cxx ’)10

conf . check_tool ( ’ node_addon ’)

1112# Library for EPICS

13

conf . env . CPPPATH = [ ’/ usr / local / epics / base / include ’, ’/ usr / local / epics / base /14

conf . check_cxx ( lib = ’ cas ’ , libpath = ’/ usr / local / epics / base / lib / linux - x86 ’)

include / os / Linux ’]
15
16

def build ( bld ) :17

obj = bld . new_task_gen ( ’ cxx ’ , ’ shlib ’ , ’ node_addon ’)

18

obj . cxxflags = [ " -g " , " - D _F I LE _ O F FS E T_ BI T S =64 " ," - D _L AR G EF I L E _ S O UR C E " , " - Wall " ]1920

obj . target = ’ nodeca ’

21

obj . source = " nodeca . cc "22

obj . libpath = ’/ usr / local / epics / base / lib / linux - x86 ’

23

obj . lib = ’ cas ’Listing B.2: Makefile to build as Node.js add-on.

B.1.3
1

ca subscription.js/*!

2* ca_subscription . js

3

* Copyright ( c ) 2012 A . UCHIYAMA in Sokendai University ( KEK ) < akito12@gmail . com >4* MIT Licensed

5
6*/// Required Node . js v0 .6.11 or later78

var pv = require ( ’ ../ build / Release / nodeca ’) ;910

var opts = require ( ’ opts ’) ;11

opts . parse ([

12

{

13

’ short ’: ’p ’ ,14’ long ’: ’ pv ’ ,Appendix B. Source Code15

’ description ’: ’ EPICS record name ’ ,16’ value ’: true ,’ required ’: true

17
18

},

19

{

20

’ short ’: ’t ’ ,21’ long ’: ’ time ’ ,22’ description ’: ’ Monitor timeout ’ ,23’ value ’: true ,’ required ’: false

24

}

25
26

102

]) ;

27
28

var record_name = opts . get ( ’ pv ’)29

var timeout = opts . get ( ’ time ’) ||030
31

pv . callback = function ( v ) {32

val = v . split ( " ," ) ;33

process . send ( val [1]) ;

34

}

35

pv . camonitor ( record_name , timeout ) ;Listing B.3: Use for event-driven function (caMonitor) using child process.

1

B.2

WebSocket Server for EPICS CA

B.2.1

app.js/*!

2* nodeca_app . js

3

* Copyright ( c ) 2012 A . UCHIYAMA in Sokendai University ( KEK ) < akito12@gmail . com >4* MIT Licensed

5
6*/// Required Node . js v0 .6.11 or later

7
8

/**

9

* Module dependencies .

10

*/

11

var express = require ( ’ express ’) ;12

var socketio = require ( ’ socket . io ’) ;13

var fs = require ( ’ fs ’) ;14

var cp = require ( ’ child_process ’) ;15

var opts = require ( ’ opts ’) ;16

var log4js = require ( ’ log4js ’) ;17
18

var use_SSL ;19

opts . parse ([

20

{

21

’ short ’: ’s ’ ,22’ long ’: ’ ssl ’ ,23’ description ’ : ’ use SSL connection ’ ,Appendix B. Source Code24

’ value ’25’ required ’

26

},

27

{

: false ,: false28’ short ’: ’p ’ ,29’ long ’: ’ port ’ ,30’ description ’ : ’ port number ’ ,31’ value ’32’ required ’: false}33
34: true ,

]) ;

35
36
37

/**

38

* EPICS CA

39

*/

40

var pv = require ( " ./ build / Release / nodeca ") ;

41
42

/**

43

* SSL key

44

*/

45

var SSL_KEY = ’ privatekey . pem ’;46

var SSL_CERT = ’ certificate . pem ’;

47
48

/**

49

* Set Express

50

*/

51

var PORT = opts . get ( ’ port ’) || 9000;

52

var HOME_DIR = ’/ public ’;

53
54

var use_SSL = opts . get ( ’ ssl ’) || 0;55

if ( use_SSL == 1) {var app = express . createServer ({56

key : fs . readFileSync ( SSL_KEY ) . toString () ,57

cert : fs . readFileSync ( SSL_CERT ) . toString ()

58

}

59
60

, express . basicAuth ( authorize )61, express . logger (), express . static ( __dirname + HOME_DIR ) ) ;62

console . log ( " Info : SSL is used . " ) ;63
64} else {var app = express . createServer (65

express . logger ()66, express . static ( __dirname + HOME_DIR ) ) ;

67
68

};

69
70

/**

71

* Output access log

72

*/

73
74

app . configure ( function () {75

log4js . configure ({76’ appenders ’: [77

{ ’ type ’: ’ console ’ } ,

78

{

103

Appendix B. Source Code

104

79

’ type ’: ’ file ’ ,80’ filename ’: ’/ mnt / websocketlogs / access . logs ’,81’ maxLogSize ’: 1024 * 1024 ,82’ backups ’: 5 ,’ category ’: [ ’ project - name ’ , ’ console ’ ] ,

83

},

84

],

85

’ replaceConsole ’: true

86

}) ;

87
88
89

var logger = log4js . getLogger ( ’ project - name ’) ;90

app . use ( log4js . connectLogger ( logger , {91’ level ’: log4js . levels . DEBUG ,92’ nolog ’: [ ’ \\ css ’ , ’ \\. js ’ , ’ \\. gif ’ ] ,’ format ’: ’: remote - addr - - ": method : url HTTP /: http - version " : status :93

content - length ": referrer " ": user - agent " ’}) ) ;

94

app . use ( app . router ) ;

95
96

}) ;

97
98
99

/**

100

* Basic authentication

101

*/

102

function authorize ( username , password ) {return ’ admin ’ === username & ’ admin ’ === password ;

103
104

}

105
106

// Open server port

107

app . listen ( PORT ) ;

108
109

var io = socketio . listen ( app , {110// log level in socket . io111// 0 : Error

112// 1 : warn

113// 2 : info

114// 3 : debug

115’ log level ’ : 1 ,116

key : fs . readFileSync ( SSL_KEY ) . toString () ,117

cert : fs . readFileSync ( SSL_CERT ) . toString ()

118

}) ;

119
120

var sublist = [];// for caSubscription

121

var intlist = [];// for caGet_interval

122

var interval_Idlist = []; // for intervalId of caGe t_interval

123

var client_info = [];124

var sock_names = [];// for IPaddr : port of clients

125
126

var c a S u b s c r i p t i o n _ c o u n t = 0;127

var c a G e t _ i n t e r v a l _ c o u n t = 0;128

var t o t a l _ c o n n e c t i o n _ c o u n t = 0;

129
130

var c on n ec t io n_ a dd re s s ;131// var d i sc o n n ec t _a dd r es s ;132

Appendix B. Source Code133
134

fs . readdirSync ( __dirname + HOME_DIR ) . forEach ( function ( filename ) {function stat ( filename ) {var stats = fs . statSync ( __dirname + HOME_DIR + ’/ ’ + filename ) ;135

return stats ;

136
137

105

}

138
139

stats = stat ( filename ) ;

140
141

// Directory check & Files (. js , . html , . htm ) check

142

extension1 = ’. js ’;143extension2 = ’. html ’;

144

extension3 = ’. htm ’;

145function check_extension ( filename ) {146

if ( filename . lastIndexOf ( extension1 ) == -1

147

&&

filename . lastIndexOf ( extension2 ) == -1 &&148

filename . lastIndexOf ( extension3 ) == -1

149

){

150

return -1

151} else {152

if ( filename . lastIndexOf ( ’ jquery ’) == -1) {153

return 1

154} else {155

return -1

156

}

157

}

158

}

159
160
161

if ( stats . isDirectory () ) {162

dirname = filename ;sock_names . push ( dirname ) ;163

stats = stat ( dirname ) ;

164
165

if ( stats . isDirectory () ) {166

fs . readdirSync ( __dirname + HOME_DIR + ’/ ’+ dirname ) . forEach ( function (167

filename ) {168

if ( check_extensi on ( filename ) == 1) {169

path_name = dirname ;path_name += " / " ;170

path_name += filename . substr (0 , filename . lastIndexOf ( ’. ’) ) ;

171
172

sock_names . push ( path_name ) ;

173

}
}) ;

174

} else {175

if ( check_extension ( filename ) == 1) {176

sock_names . push ( filename . substr (0 , filename . lastIndexOf ( ’. ’) ) ) ;

177

}

178
179

}

180

} else {if ( check_extension ( filename ) == 1) {181

sock_names . push ( filename . substr (0 , filename . lastIndexOf ( ’. ’) ) ) ;

182

}

183
184

}

185

}) ;

186

sock_names = unique ( sock_names )Appendix B. Source Code

106

187
188

/**

189

* Set socket_main function

190

*/

191

for ( i = 0; i < sock_names . length ; i ++) {192

sock_name = sock_names [ i ];193

io . of ( ’/ ’ + sock_names [ i ]) . on ( ’ connection ’ , socket_main ) ;console . log ( " Set NameSpace " + i

194+ " : " + sock_name ) ;

195

}

196

io . of ( ’ ’) . on ( ’ connection ’, socket_main ) ;

197
198
199
200
201
202

function socket_main ( socket , data ) {/*** Store the information of connected clients*/c on n ec ti o n_ a d d re s s = socket . handshake . address ;

203
204

client_info [ t o t a l _ c o n n e c t i o n _ c o u n t ] = " " ;205client_info [ t o t a l _ c o n n e c t i o n _ c o u n t ] += c o n n e c t i o n _ a d d r e s s . address ;206

client_info [ t o t a l _ c o n n e c t i o n _ c o u n t ] += " : " ;207client_info [ t o t a l _ c o n n e c t i o n _ c o u n t ] += c o n n e c t i o n _ a d d r e s s . port ;208

client_info [ t o t a l _ c o n n e c t i o n _ c o u n t ] += " ," ;209

client_info [ t o t a l _ c o n n e c t i o n _ c o u n t ] += socket . id ;210

client_info [ t o t a l _ c o n n e c t i o n _ c o u n t ] += " ," ;211

client_info [ t o t a l _ c o n n e c t i o n _ c o u n t ] += socket . namespace . name ;

212
213

console . log ( " New connection from " + client_info [ t o t a l _ c o n n e c t i o n _ c o u n t ]) ;214

t o t a l _ c o n n e c t i o n _ c o u n t ++;

215
216

socket . on ( ’ caPut ’ , function ( data ) {caPut ( socket , data ) ;

217
218

}) ;

219
220
221

socket . on ( ’ caGet ’ , function ( data ) {process . nextTick (function () { caGet ( socket , data ) }

222
223
224

);
}) ;

225
226
227

socket . on ( ’ caGet_interval ’ , function ( data ) {process . nextTick (function () { caGet_interval ( socket , data ) }

228
229
230

);
}) ;

231
232
233

socket . on ( ’ caMonitor ’ , function ( data ) {process . nextTick (234

function () { caMonitor ( socket , data ) }

235

);

236

}) ;

237
238

/*******************************************************/

239/* Enable under here if use Operator Intervening System */240/*******************************************************/

241

socket . on ( ’ RequestOpe ’ , function ( data ) {Appendix B. Source Codevar n = cp . fork (

242
243

__dirname + ’/ module / request ’,244[ ’ -r ’ , data . pv , ’ -u ’ , process . env . USER ] ,{ env : process . env , timeout :" 1 " }

245
246

);

247

}) ;

248

socket . on ( ’ OpeInfo ’ , function ( data ) {249

var n = cp . fork (

250
251

__dirname + ’/ module / ope_info ’,252[ ’ -r ’ , data . pv , ’ -u ’ , process . env . USER , ’ -s ’ , data . st ] ,{ env : process . env , timeout :" 1 " }

253

);

254
255

if (! data . event ) {256

data . event = " ca " ;

257

}

258
259

n . on ( ’ message ’ , function ( m ) {260

var obj = {};

261
262

obj [ data . data_name ] = m ;263

socket . volatile . emit ( data . event , obj ) ;socket . volatile . broadcast . emit ( data . event , obj ) ;

264

}) ;

265
266

setTimeout ( function () {267

process . kill ( n . pid , ’ SIGHUP ’) ;268

} ,1000) ;

269

}) ;

270
271

/********************************************************/

272

socket . on ( ’ caPut_counter ’ , function ( data ) {273

caPut_counter ( socket , data ) ;

274

}) ;

275
276
277

socket . on ( ’ disconnect ’ , function () {278

var disconnect_sid = socket . id ;279

console . log ( " Disconnect : " + disconnect_sid ) ;280

process . nextTick (281

function () { do_disconnect ( disconn ect_sid ) }
);

282

}) ;

283
284

}

285
286
287

function caGet ( socket , data ) {try {v = pv . caget ( data . pv ) ;

288
289

} catch ( e ) {v = " undefined " ;290console . log ( " caGet : " + data . pv + " : " + e ) ;

291
292

}

293

if (! data . event ) {data . event = " ca " ;

294
295

}

296

if ( typeof obj === " undefined " ) {107

Appendix B. Source Codevar obj = {};

297
298

}

299

obj [ data . data_name ] = v ;300

socket . volatile . emit ( data . event , obj ) ;socket . volatile . broadcast . emit ( data . event , obj ) ;

301
302

}

303
304

function caPutCallback ( socket , data ) {
try {

305

v = pv . caget ( data . callback_pv ) ;306} catch ( e ) {

307
308

v = " undefined " ;309

console . log ( " caGet Error : " + e ) ;

310

}

311

if (! data . event ) {data . event = " ca " ;

312
313

}

314

if ( typeof obj === " undefined " ) {var obj = {};

315
316

}

317

obj [ data . callback_obj ] = v ;318

socket . volatile . emit ( data . event , obj ) ;socket . volatile . broadcast . emit ( data . event , obj ) ;

319
320

}

321
322

function caFailCallback ( socket , data ) {if (! data . event ) {323

data . event = " ca " ;

324
325

}

326

v = " caPut_failuer " ;327

if ( typeof obj === " undefined " ) {var obj = {};

328
329

}

330

obj [ data . callback_obj ] = v ;331socket . volatile . emit ( data . event , obj ) ;socket . volatile . broadcast . emit ( data . event , obj ) ;

332
333

}

334
335

function ca Suc ces s Ca l lba ck ( socket , data ) {if (! data . event ) {336

data . event = " ca " ;

337
338

}

339

v = " caPut_Success " ;340

if ( typeof obj === " undefined " ) {var obj = {};

341
342

}

343

obj [ data . callback_obj ] = v ;344

socket . volatile . emit ( data . event , obj ) ;socket . volatile . broadcast . emit ( data . event , obj ) ;

345
346

}

347
348
349
350
351

function caPut ( socket , data ) {process . nextTick ( function () {try {pv . caput ( data . pv , data . val ) ;108

Appendix B. Source Code352

if ( data . callback_pv ) {353

caPutCallback ( socket , data ) ;

354
355

}

356

if ( data . data_name ) {setTimeout ( caGet ( socket , data ) ,1000) ;

357
358

}

359

if ( data . callback_obj && ! data . callback_pv ) {ca Suc cess Cal lba ck ( socket , data ) ;

360
361

}

362

} catch ( e ) {if ( data . callback_obj ) {363

caFailCallback ( socket , data ) ;

364

}

365

console . log ( " caPut : " + data . pv + " : " + e ) ;

366

}

367

}) ;

368
369

}

370
371
372

function caMonitor ( socket , data ) {
/**

373

* Check the sublist array for caSubscription function374* run_flag = 0; Run caSubscription

375* run_flag = 1; Already run
*/

376
377

if (! data . timeout ) {data . timeout = " 0 " ;

378
379

}

380

if (! data . event ) {data . event = " ca " ;

381
382

}

383
384

if ( data . timeout == " 0 " ) {385

str_sublist = " " ;386

str_sublist += data . event ;387

str_sublist += " ," ;388

str_sublist += data . pv ;389

str_sublist += " ," ;390

str_sublist += data . timeout ;391

str_sublist += " ," ;392

str_sublist += data . data_name ;393

str_sublist += " ," ;str_sublist += socket . namespace . name ;

394
395

} else {var PID_id = caSubscription ( socket , data ) ;

396
397

}

398
399

run_flag = 0;

400

for ( j = 0; j < sublist . length ; j ++) {401

if ( sublist [ j ]. indexOf ( str_sublist ) !== -1) {402

run_flag = 1;403

sublist [ j ] += " ," ;404

sublist [ j ] += socket . id ;405

console . log ( " Already running : caMonitor : " + j + " : " + sublist [ j ]) ;406

break ;

109

Appendix B. Source Code
}

407

}

408
409

if ( run_flag == 0) {

410
411

var PID_id = caSubscription ( socket , data ) ;412// Concatennate413

str_sublist += " ," ;414

str_sublist += PID_id ;415

str_sublist += " | " ;1. pv , 2. timeout , 3. data_name , 4. name_space , 5. PID_id416

str_sublist += socket . id ;417

sublist [ c a S u b s c r i p t i o n _ c o u n t ] = str_sublist ;console . log ( " caMonitor : " + c a S u b s c r i p t i o n _ c o u n t + " : " + sublist [418

c a S u b s c r i p t i o n _ c o u n t ]) ;c a S u b s c r i p t i o n _ c o u n t ++;

419

}

420
421

}

422
423

function caSubscription ( socket , data ) {var n = cp . fork (

424
425

__dirname + ’/ module / ca_su b sc ri pt i on ’,426[ ’ -p ’ , data . pv , ’ -t ’ , data . timeout ] ,{ env : process . env , timeout : data . timeout }

427

);

428
429

if ( data . timeout != " 0 " ) {430

setTimeout (431

function () {432

stop_caMonitor ( n . pid ) ;

433

},

434

data . timeout * 1000

435

);

436

}

437
438

if (! data . event ) {439

data . event = " pv " ;

440

}

441
442

var sid = socket . id ;443

n . on ( ’ message ’ , function ( m ) {444

var obj = {};445

obj [ data . data_name ] = m ;446

socket . volatile . emit ( data . event , obj ) ;socket . volatile . broadcast . emit ( data . event , obj ) ;

447

}) ;

448

return n . pid ;

449
450

}

451
452

function caGet_interval ( socket , data ) {

453
454

if (! data . int_sec ) {455

data . int_sec = 1;456

var interval_time =1000;457} else {var interval_time = data . int_sec * 1000;

458
459
460

}

110

Appendix B. Source Code461

if (! data . event ) {data . event = " ca " ;

462
463

}

464
465

str_intlist = " " ;466

str_intlist += data . event ;467

str_intlist += " ,"468

str_intlist += data . pv ;469

str_intlist += " ," ;470

str_intlist += data . int_sec ;471

str_intlist += " ," ;472

str_intlist += data . data_name ;473

str_intlist += " ," ;474

str_intlist += socket . namespace . name ;

475
476

var event = data . event ;

477
478

function caget_int () {
try {

479

v = pv . caget ( data . pv ) ;480} catch ( e ) {

481
482

console . log ( " caGet : " + data . pv + " : " + e ) ;483

v = ’ undefined ’;

484

}

485

var obj = {};486

obj [ data . data_name ] = v ;487

socket . volatile . emit ( data . event , obj ) ;socket . volatile . broadcast . emit ( data . event , obj ) ;

488
489

}

490
491

run_flag = 0;492

for ( j = 0; j < intlist . length ; j ++) {493

if ( intlist [ j ]. indexOf ( str_intlist ) !== -1) { // Not include == -1

494
495

run_flag = 1;

496

intlist [ j ] += " ," ;497

intlist [ j ] += socket . id ;498

console . log ( " Already running : caGet_interval : " + j + " : " + intlist [ j ]) ;499

break ;
}

500
501
502

}

503
504

if ( run_flag == 0) {interval_Id = setInterval ( caget_int , interval_time ) ;

505
506

interval_Idlist [ c a G e t _ i n t e r v a l _ c o u n t ] = interval_Id ;507

str_intlist += " ," ;508

str_intlist += " | " ;509

str_intlist += socket . id ;510

intlist [ c a G e t _ i n t e r v a l _ c o u n t ] = str_intlist ;console . log ( " caGet_interv al : " + c a G e t _ i n t e r v a l _ c o u n t + " : " + intlist [511

c a G e t _ i n t e r v a l _ c o u n t ]) ;c a G e t _ i n t e r v a l _ c o u n t ++;

512
513

}

514

return interval_Id ;111

Appendix B. Source Code
515

}

516
517

function do_disconnect ( disconnect_sid ) {

518
519

client_sid = [];520

for ( j = 0; j < sublist . length ; j ++) {str_sid = " " ;

521
522

client_sids = sublist [ j ]. split (" | " ) ;523

if ( client_sids [1]) {client_sid = client_sids [1]. split (" ," ) ;

524

}

525

for ( m = 0; m < client_sid . length ; m ++) {526

if ( client_sid [ m ] !== disconnect_sid && client_sid [ m ] !== " " ) {527

str_sid += client_sid [ m ];528

str_sid += " ," ;

529

}

530

}

531
532

if ( sublist [ j ]) {

533
534

client_sids = sublist [ j ]. split (" | " ) ;535

sublist [ j ] = client_sids [0];536sublist [ j ] += " | " ;537

sublist [ j ] += str_sid ;538

if (! str_sid ) {539// pv_data : 0. event , 1. pv , 2. timeout , 3. data_name , 4. PID_id

540
541

pv_data = client_sids [0]. split (" ," ) ;542

stop_caMonitor ( pv_data [5]) ;sublist [ j ] = " " ;

543

}

544

}

545
546

}

547
548

str_sid = " " ;549

client_sid = [];550

for ( j = 0; j < intlist . length ; j ++) {551

client_sids = intlist [ j ]. split (" | " ) ;552

if ( client_sids [1]) {client_sid = client_sids [1]. split (" ," ) ;

553

}

554
555

}

556
557

for ( j = 0; j < client_sid . length ; j ++) {if ( client_sid [ j ] !== disconnec t_sid && client_sid [ j ] !== " " ) {558

str_sid += client_sid [ j ];559

str_sid += " ," ;

560

}

561
562

}

563
564
565

for ( j = 0; j < intlist . length ; j ++) {if ( intlist [ j ]) {566

client_sids = intlist [ j ]. split (" | " ) ;567

intlist [ j ] = client_sids [0];568intlist [ j ] += " | " ;569

intlist [ j ] += str_sid ;112

Appendix B. Source Code570

if (! str_sid ) {571// pv_data : 0. event , 1. pv , 2. timeout , 3. data_name , 4. intervalId

572
573

s t o p _ ca G e t _ i n t e rv a l ( i nt er va l_I dl is t [ j ]) ;574

intlist [ j ] = " " ;
}

575

}

576

}

577
578

}

579
580

function stop_caMonitor ( PID_id ) {581

process . kill ( PID_id , ’ SIGHUP ’) ;582

console . log ( " LOG : child_process . kill : " + PID_id ) ;return 0;

583
584

}

585
586

function s t op _ c a G e t _ i nt e r v a l ( Id_obj ) {587

clearInterval ( Id_obj ) ;588

console . log ( " LOG : clearInterval " ) ;return 0;

589
590

}

591
592

function caPut_counter ( socket , data ) {
try {

593

v = pv . caget ( data . pv ) ;594} catch ( e ) {

595
596

console . log ( " caGet : " + data . pv + " : " + e ) ;597

v = ’ undefined ’;return 0

598
599

}

600

try {
if ( v !== null && v !== undefined && ! isNaN ( v ) ) {601

pv . caput ( data . pv , v + data . val ) ;

602

}

603

if ( data . callback_pv ) {604

caPutCallback ( socket , data ) ;

605
606

}

607

} catch ( e ) {if ( data . callback_pv ) {608

caFailCallback ( socket , data ) ;

609

}

610

console . log ( " caPut_counter : " + data . pv + " : " + e ) ;

611

}

612
613

}

614
615

function unique ( array ) {616

var storage = {};617

var uniqueArray = [];618

var i , value ;619

for ( i =0; i < array . length ; i ++) {620

value = array [ i ];621

if (!( value in storage ) ) {622

storage [ value ] = true ;uniqueArray . push ( value ) ;

623
624

}

113

Appendix B. Source Code}625

return uniqueArray ;

626
627

114

}

Listing B.4:

WebSocket Server coded in Node.js and Socket.IO (server-sidejavascript).

1
2

B.3

Example of WebSocket Client for EPICS CA

B.3.1

index.html< html >< head >3< title > ca - WebSocket example page </ title >4< link rel = " stylesheet " type = " text / css " href = " index . css " / >5< script type = " text / javascript " src = " / socket . io / socket . io . js " > </ script >6< script type = " text / javascript " src = " / flot / jquery . js " > </ script >7< script type = " text / javascript " src = " ./ index . js " > </ script >8</ head >9< body >10<h4 > Record information with realtime update ( camonitor written in Node . js ) </
h4 >

11

< div id = " EPICS2 " >12<p > PV : akito12Host : ai1 ( caMonitor ) : < span id = " TEST_record1 " > </p >13<p > PV : akito12Host : aiExample ( caMonitor ) : < span id = " TEST_record2 " > </p ><p > PV : akito12Host : aiExample1 ( caGet ) : < span id = " TEST_record3 " > </p >1415</ div >16
17< div id = " EPICS " ><p > PV : akito12Host : xxxExample < span id = " TEST_record4 " > </p >18<li > < a id = " caput - update1 " href = " # " onclick = " caput_update1 () " > caPut

19akito12Host : xxxExample 1 </a >
<br >

20

<li > < a id = " caput - update2 " href = " # " onclick = " caput_update2 () " > caPut

21akito12Host : xxxExample 0 </a >22</ div >23< div id = " CALLBACK " ><p > CALLBACK : < span id = " LOG " > </p >

24
25
26
27

</ div ></ body ></ html >Listing B.5: Example of WebSocket Client for EPICS Channel Access (html file)

B.3.2

index.js

1

var socket = io . connect ( ’/ index ’) ;2

function callback_func ( obj , msg ) {Appendix B. Source Code$ ( obj ) . html ( msg ) ;

3
4

115

}

5
6

socket . emit ( ’ caMonitor ’ ,{ pv : ’ rootHost : aiExample1 ’, data_name : ’ TEST_record1 ’ }) ;7

socket . emit ( ’ caMonitor ’ ,{ pv : ’ rootHost : aiExample1 ’, data_name : ’ TEST_record2 ’ }) ;8

socket . emit ( ’ caGet ’ ,{ pv : ’ rootHost : aiExample1 ’, data_name : ’ TEST_record3 ’ }) ;9

caput_update1 = function () {socket . emit ( ’ caPut ’ ,{10
11

pv : ’ rootHost : xxxExample ’ ,12val : 199 ,13

data_name : ’ TEST_record4 ’ ,14callback_pv : ’ rootHost : aiExample2 ’,

15callback_obj : ’ TEST_callback1 ’} , callback_func ( ’# LOG ’ , ’ < font color = red > Data set 1 !! </ font > ’) ) ;

16
17

}

18
19

caput_update2 = function () {socket . emit ( ’ caPut ’ ,{20
21

pv : ’ rootHost : xxxExample ’ ,22val : 0 ,23

data_name : ’ TEST_record4 ’ ,24// callback_pv : ’ rootHost : aiExample2 ’,callback_obj : ’ TEST_callback1 ’

25} , callback_func ( ’# LOG ’ , ’ < font color = green > Data set 0 .. </ font > ’) ) ;

26
27

}

28
29
30

socket . on ( ’ ca ’ , function ( data ) {31$ ( ’# TEST_record1 ’) . html ( data . TEST_record1 ) ;32$ ( ’# TEST_record2 ’) . html ( data . TEST_record2 ) ;33$ ( ’# TEST_record3 ’) . html ( data . TEST_record3 ) ;34$ ( ’# TEST_record4 ’) . html ( data . TEST_record4 ) ;if ( data . TEST_callback1 !== undefined ) console . log ( " !! " + data . TES T_callback1

35+ " !! " ) ;
36

}) ;

Listing B.6: Example of WebSocket Client for EPICS Channel Access (client-sidejavascript)

1

B.4

Operator Intervention System

B.4.1

accept.js/*!

2* accept . js

3

* Copyright ( c ) 2013 A . UCHIYAMA in Sokendai University ( KEK ) < akito12@gmail . com >4* MIT Licensed

5

*

6

*/

7
8

/**

Appendix B. Source Code9

* Module dependencies .

10

*/

11

var mysql = require ( ’ mysql ’) ;12

var conf = require ( ’ ./ conf ’) ;13

var opts = require ( ’ opts ’) ;14
15

opts . parse ([{16’ short ’17: ’v ’ ,18’ long ’19’ description ’ : ’ Show version and exit ’ ,’ callback ’

20
21

},

22

{

: ’ version ’ ,: function () { console . log ( ’ v1 .0 ’) ; process . exit (1) ; }23’ short ’24’ long ’: ’ id ’ ,25’ description ’: ’ record id for MySQL table ’,’ value ’26: ’i ’ ,: true ,’ required ’: true30’ short ’: ’o ’ ,31’ long ’: ’ operator ’ ,32’ description ’: ’ Set operator name ’ ,

27
28

},

29

{

’ value ’33: true},35
36: true ,’ required ’

34

/*
{

37

’ short ’38: ’r ’ ,39’ long ’: ’ pv ’ ,40’ description ’: ’ request EPICS record ’ ,41’ value ’: true ,42’ required ’: false46’ short ’: ’t ’ ,47’ long ’: ’ time ’ ,48’ description ’: ’ reserved time ’ ,

},

43
44

*/
{

45

49

’ value ’: false}51
52: true ,’ required ’

50

]) ;

53
54

var connection = mysql . crea teC o n n e c t i o n ({

55

host

: conf . MYSQL_HOST ,

56

user

: conf . MYSQL_USER ,57

password : conf . MYSQL_PASS ,58

database : conf . DATABASE ,59
60

insecureAuth : true
}) ;

61
62

var TABLE = ’ ope_request ’;63

var _id = opts . get ( ’ id ’) ;116

Appendix B. Source Code64

117

var _ope = opts . get ( ’ operator ’) ;65
66

if (! opts . get ( ’ time ’) ) {var _time = 120;

67
68

} else {var _time = opts . get ( ’ time ’) ;

69
70

}

71
72

console . log ( " Record id : " + _id ) ;73

console . log ( " Operator : " + _ope ) ;74

console . log ( " Reserved time : " + _time ) ;75
76

var nowdate = new Date () ;77
78

var post = {ope_name : _ope ,7980

ope_allowed_time : nowdate ,81

reserved_sec : _time ,status : 1

82
83

}

84

connection . query (85’ UPDATE ’ + TABLE + ’ SET ? where id = ’ + _id + ’ and status = 0 ’ ,86post ,

87

function ( err , result ) {if ( err ) {88
89

console . log ( ’ database update error ’) ;90

console . log ( err ) ;connection . destroy () ;91

} else {92

console . log ( result ) ;93

connection . end () ;

94

}

95

}

96
97

);

Listing B.7: Insert accept command into MySQL-based database for operation with
output.

B.4.2
1

conf.jsmodule . exports = {2

MYSQL_HOST : ’RDB - srv ’ ,3

DATABASE : ’ access_data ’ ,4

TABLE : ’ ope_request ’ ,5

MYSQL_USER : ’ xxx ’ ,MYSQL_PASS : ’ xxxxxx ’

6
7

};

B.4.3

end.jsAppendix B. Source Code

1

118

/*!

2

* end . js

3

* Copyright ( c ) 2013 A . UCHIYAMA in Sokendai University ( KEK ) < akito12@gmail . com >4* MIT Licensed

5

*

6

*/

7
8

/**

9

* Module dependencies .

10

*/

11

var mysql = require ( ’ mysql ’) ;12

var conf = require ( ’ ./ conf ’) ;13

var opts = require ( ’ opts ’) ;14
15

opts . parse ([{16’ short ’17: ’v ’ ,18’ long ’19’ description ’ : ’ Show version and exit ’ ,’ callback ’

20
21

},

22

{

: ’ version ’ ,: function () { console . log ( ’ v1 .0 ’) ; process . exit (1) ; }23’ short ’24’ long ’: ’ id ’ ,25’ description ’: ’ record id for MySQL table ’,’ value ’26: ’i ’ ,: true ,’ required ’: true30’ short ’: ’o ’ ,31’ long ’: ’ operator ’ ,32’ description ’: ’ Set operator name ’ ,

27
28

},

29

{

’ value ’33: true},35
36: true ,’ required ’

34

/*
{

37

’ short ’38: ’r ’ ,39’ long ’: ’ pv ’ ,40’ description ’: ’ request EPICS record ’ ,41’ value ’: true ,42’ required ’: false46’ short ’: ’t ’ ,47’ long ’: ’ time ’ ,48’ description ’: ’ reserved time ’ ,

},

43
44

*/
{

45

49

’ value ’’ required ’

50: false}51
52: true ,

]) ;

53
54

var connection = mysql . crea teC o n n e c t i o n ({Appendix B. Source Code
55

host

: conf . MYSQL_HOST ,

56

user

: conf . MYSQL_USER ,57

password : conf . MYSQL_PASS ,58

database : conf . DATABASE ,insecureAuth : true

59
60

119

}) ;

61
62

var TABLE = ’ ope_request ’;63

var _id = opts . get ( ’ id ’) ;64

var _ope = opts . get ( ’ operator ’) ;65
66

if (! opts . get ( ’ time ’) ) {var _time = 120;

67
68

} else {var _time = opts . get ( ’ time ’) ;

69
70

}

71
72

console . log ( " Record id : " + _id ) ;73

console . log ( " Operator : " + _ope ) ;74
75

var nowdate = new Date () ;76
77

var post = {78

ope_endtime : nowdate ,79

status : 2

80

}

81
82

connection . query (83’ UPDATE ’ + TABLE + ’ SET ? where id = ’ + _id + ’ and status = 1 and ope_name84

post ,

85

function ( err , result ) {= \ ’ ’+ _ope + ’\ ’ ’ ,if ( err ) {86
87

console . log ( ’ database update error ’) ;88

console . log ( err ) ;connection . destroy () ;89

} else {90

console . log ( result ) ;91

connection . end () ;

92

}

93

}

94
95

);

Listing B.8: Insert end command into MySQL-based database for timeout.

B.4.4
1

request.js/*!

2* request . js

3

* Copyright ( c ) 2013 A . UCHIYAMA in Sokendai University ( KEK ) < akito12@gmail . com >4* MIT Licensed

5

*

6

*/

Appendix B. Source Code

120

7
8

/**

9

* Module dependencies .

10

*/

11

var mysql = require ( ’ mysql ’) ;12

var conf = require ( ’ ./ conf ’) ;13

var opts = require ( ’ opts ’) ;14
15

opts . parse ([
{

16
17

’ short ’18’ long ’: ’ version ’ ,19’ description ’: ’ Show version and exit ’ ,’ callback ’: function () { console . log ( ’ v1 .0 ’) ; process . exit (1) ; }23’ short ’: ’u ’ ,24’ long ’: ’ user ’ ,25’ description ’: ’ Set user name ’ ,

20
21

},

22

{

’ value ’26: ’v ’ ,: true ,’ required ’: true30’ short ’: ’r ’ ,31’ long ’: ’ pv ’ ,32’ description ’: ’ request EPICS record ’ ,

27
28

},

29

{

’ value ’33’ required ’

34: true}35
36: true ,

]) ;

37
38

var connection = mysql . crea teC o n n e c t i o n ({

39

host

: conf . MYSQL_HOST ,

40

user

: conf . MYSQL_USER ,41

password : conf . MYSQL_PASS ,42

database : conf . DATABASE ,insecureAuth : true

43
44

}) ;

45
46

var TABLE = ’ ope_request ’;

47

var _user = opts . get ( ’ user ’) ;48

var _pv = opts . get ( ’ pv ’) ;49
50

console . log ( " User : " + _user ) ;51

console . log ( " PV : " + _pv ) ;52

var nowdate = new Date () ;53
54

var post = {user_name : _user ,55
56

request_record : _pv ,57

request_time : nowdate ,status : 0

58
59

}

60
61

connection . query (Appendix B. Source Code

121

62

’ INSERT INTO ’ + TABLE + ’ SET ? ’ ,63post ,

64

function ( err , result ) {if ( err ) {65
66

console . log ( ’ database insert error ’) ;67

console . log ( err ) ;connection . destroy () ;68

} else {69

console . log ( result ) ;70

connection . end () ;

71

}

72

}

73
74

);

Listing B.9:

Insert request command into MySQL-based database for output
operation request.

B.4.5

pooling.js

1

var mysql = require ( ’ mysql ’) ;2

var conf = require ( ’ ./ conf ’) ;3
4

var pool = mysql . createPool ({

5

host

: conf . MYSQL_HOST ,

6

user

: conf . MYSQL_USER ,7

password : conf . MYSQL_PASS ,8

database : conf . DATABASE ,9

insecureAuth : true

10

}) ;

11

var TABLE = ’ ope_request ’;

12
13
14

pool . getConnection ( function ( err , connection ) {connection . query ( ’ select * from ’ + TABLE , function ( err , result , fields ) {15

if ( err ) throw err ;16else {
console . log ( ’ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ’) ;17

for ( var i in result ) {18

var val = result [ i ];19

console . log (20
21

val . id + ’: ’22

+ val . user_name + ’: ’23

+ val . request_time + ’: ’24

+ val . status

25

);
}

26

}

27

connection . end () ;

28
29
30

}) ;
}) ;

Listing B.10: Check the operation flag in MySQL-based database.

Appendix B. Source Code

B.4.6

122

main.php1<? php2/*

2013 -07 -09 [ A . UCHIYAMA ] */3/*

file :

4require ( " ./ session_check . php ") ;5

main . php

*/

?>

6
7

<! DOCTYPE html >8< html lang = " ja " >9< head >10< meta charset = " utf -8 " >11< title > Operator Intervening System & middot ; Powered by Bootstrap </ title >12< meta name = " viewport " content = " width = device - width , initial - scale =1.0 " >13< meta name = " description " content = " " >14< meta name = " author " content = " A . UCHIYAMA " >1516<! - - Le styles -->17< link href = " ../ docs / assets / css / bootstrap . css " rel = " stylesheet " >18< style type = " text / css " >19body {
padding - top : 20 px ;20

padding - bottom : 60 px ;

21
22

}

23
24

/* Custom container */25. container {26

margin : 0 auto ;27

max - width : 1000 px ;

28

}

29

. container > hr {margin : 60 px 0;

30
31

}

32
33

/* Main marketing message and sign up button */34. jumbotron {35

margin : 80 px 0;36

text - align : center ;

37

}

38

. jumbotron h1 {39

font - size : 100 px ;40line - height : 1;

41

}

42

. jumbotron . lead {43

font - size : 24 px ;line - height : 1.25;

44
45

}

46

. jumbotron . btn {font - size : 21 px ;47

padding : 14 px 24 px ;

48
49

}

50
51

/* Supporting marketing content */52. marketing {53

margin : 60 px 0;

Appendix B. Source Code
54

}

55

. marketing p + h4 {margin - top : 28 px ;

56
57

123

}

58
59
60

/* Customize the navbar links to be fill the entire space of the . navbar */61. navbar . navbar - inner {padding : 0;

62
63

}

64

. navbar . nav {65

margin : 0;66

display : table ;width : 100%;

67
68

}

69

. navbar . nav li {70

display : table - cell ;71width : 1%;
float : none ;

72
73

}

74

. navbar . nav li a {75

font - weight : bold ;76

text - align : center ;77border - left : 1 px solid rgba (255 ,255 ,255 ,.75) ;border - right : 1 px solid rgba (0 ,0 ,0 ,.1) ;

78
79

}

80

. navbar . nav li : first - child a {border - left : 0;81border - radius : 3 px 0 0 3 px ;

82
83

}

84

. navbar . nav li : last - child a {border - right : 0;85border - radius : 0 3 px 3 px 0;

86
87

}

88

</ style >89< link href = " ../ docs / assets / css / bootstrap - responsive . css " rel = " stylesheet " >9091<! - - HTML5 shim , for IE6 -8 support of HTML5 elements -->92<! - -[ if lt IE 9] >93
94

< script src = " ../ docs / assets / js / html5shiv . js " > </ script ><![ endif ] - - >95
96<! - - Fav and touch icons -->97< link rel = " apple - touch - icon - precomposed " sizes = " 144 x144 " href = " ../ docs / assets/ ico / apple - touch - icon -144 - precomposed . png " >98< link rel = " apple - touch - icon - precomposed " sizes = " 114 x114 " href = " ../ docs / assets/ ico / apple - touch - icon -114 - precomposed . png " >99< link rel = " apple - touch - icon - precomposed " sizes = " 72 x72 " href = " ../ docs / assets/ ico / apple - touch - icon -72 - precomposed . png " >< link rel = " apple - touch - icon - precomposed " href = " ../ docs / assets

100/ ico / apple - touch - icon -57 - precomposed . png " >< link rel = " shortcut icon " href = " ../ docs / assets

101/ ico / favicon . png " >102
103</ head >Appendix B. Source Code
104

124

< body >105< div class = " container " >

106
107

< div class = " masthead " >108< h3 class = " muted " >< img src = " ../ img / logo101 . gif " width = " 50 " height = " 50 " >109

Operator Intervening System </ h3 ><p class = " text - right " >< span class = " badge " ><i class = " icon - user " > </i > Login110

Operator < strong > <? php echo $user ; ? > </ strong > </ span > </p >< div class = " navbar " >

111
112

< div class = " navbar - inner " >113< div class = " container " >< ul class = " nav " >114< li class = " active " ><a href = " ./ main . php " ><i class = " icon - cog " > </i >115

Home </ a > </ li ><li > < a href = " ./ accesslog . php " ><i class = " icon - eye - open " > </i >116

Access Log </ a > </ li ><li > < a href = " ./ putlog . php " ><i class = " icon - file " > </i > EPICS Put

117

Log </ a > </ li ><li > < a href = " ./ gwreport . php " ><i class = " icon - list - alt " > </i >118

Gateway Report </ a > </ li ><li > < a href = " ./ asg . php " ><i class = " icon - warning - sign " > </i > Access Security </ a

119

> </ li ><li > < a href = " ./ logout . php " > Logout </ a > </ li >120</ ul >

121</ div >122</ div >123</ div > <! - - /. navbar -->124</ div >125
126<! - - Example row of columns -->127< div class = " row - fluid " >128< div class = " span6 " >129<h2 > REQUEST PVs </ h2 >130

< div id = " request " > </ div ><p > < a class = " btn btn - primary " onClick = " On_all_check () " > ALL & raquo ; </a >&131nbsp ;<a class = " btn " onClick = " open_box ( ’ <? php echo $user ; ? > ’) " > NEXT </ a >&132

nbsp ;<a class = " btn " onClick = " close_box () " > CANCEL </ a >

133
134

< div id = " reserved1 " > </ div > </p >135</ div >136
137< div class = " span6 " >

138
139

<h2 > IN - OPERATION </ h2 >140

< div id = " operation " > </ div ><p > </p >141</ div >142</ div >

143
144

<br >

145

< div class = " row - fluid " >146< div class = " span10 " >147<h2 > DONE PVs </ h2 >148

< div id = " done " > </ div >149<p > </p >Appendix B. Source Code125</ div >150</ div >

151
152

<hr >

153
154
155

<! - - Modal -->156< div aria - hidden = " true " aria - labelledby = " myModalLabel " class = " modal hide fade " id
= " myModal " role = " dialog " tabindex = " -1 " >

157
158

< div id = " modal " > </ div ></ div >159
160< div class = " footer " ><p >& copy ; RIKEN Nishina Center Beam Dynamics & Diagnostics Team . and KEK

161</ p >
162</ div >163
164</ div > <! - - / container -->

165
166

<! - - Le javascript167= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = -->168<! - - Placed at the end of the document so the pages load faster -->169< script src = " ../ docs / assets / js / jquery . js " > </ script >170< script src = " ../ docs / assets / js / bootstrap - transition . js " > </ script >171< script src = " ../ docs / assets / js / bootstrap - alert . js " > </ script >172< script src = " ../ docs / assets / js / bootstrap - modal . js " > </ script >173< script src = " ../ docs / assets / js / bootstrap - dropdown . js " > </ script >174< script src = " ../ docs / assets / js / bootstrap - scrollspy . js " > </ script >175< script src = " ../ docs / assets / js / bootstrap - tab . js " > </ script >176< script src = " ../ docs / assets / js / bootstrap - tooltip . js " > </ script >177< script src = " ../ docs / assets / js / bootstrap - popover . js " > </ script >178< script src = " ../ docs / assets / js / bootstrap - button . js " > </ script >179< script src = " ../ docs / assets / js / bootstrap - collapse . js " > </ script >180< script src = " ../ docs / assets / js / bootstrap - carousel . js " > </ script >181< script src = " ../ docs / assets / js / bootstrap - typeahead . js " > </ script >

182
183
184

< script src = " ../ js / main . js " > </ script ></ body ></ html >Listing B.11: Web interface for operator intervention system.

Bibliography[1] International Committee for Future Accelerators.
work: Icfa task force reports, December 2001.

A global accelerator net-

URL http://www.fnal.gov/

directorate/icfa/icfa_tforce_reports.html.
[2] Reinhard Bacher, Philip Duval, and Stephen Herb. What are the controls requirements for the global accelerator network? Proceedings of 8th International Conference on Accelerator & Large Experimental Physics Control Systems, pages 588–
590, 2001. URL http://www.slac.stanford.edu/econf/C011127/THAP056.pdf.
[3] Deborah Agarwal, Gary Olson, and Judy Olson. Collaboration tools for the global
accelerator network: Workshop report. pages 1–15, September 2002. URL http://cds.cern.ch/record/747060/files/34081372.pdf.
[4] Silvia Gabrielli, Markus Hodapp, and Roberto Ranon. Designing a multipurpose
virtual laboratory to support communities of practice in physics. In e-Science andGrid Computing, 2006. e-Science’06. Second IEEE International Conference on,
pages 139–139. IEEE, 2006.
[5] R Pugliese, A Busato, D Favretto, F Bonaccorso, M Prica, and A Curri. Virtually
there: The control room of the future. Proceedings of ICALEPCS07 Knoxville
Tennessee USA, pages 418–420, 2007. URL http://accelconf.web.cern.ch/accelconf/ica07/PAPERS/WPPB10.PDF.
[6] Lucas Taylor, Erik Gottschalk, Kaori Maeshima, and Patricia McBride. Cms
centres for control, monitoring, offline operations and prompt analysis. In Journalof Physics: Conference Series, volume 119, page 072029. IOP Publishing, 2008.
URL http://iopscience.iop.org/1742-6596/119/7/072029.
[7] Apache tomcat. URL http://tomcat.apache.org/.
[8] Root. URL http://root.cern.ch/drupal/.
[9] Alan L Stone. Cms remote monitoring at fermilab. In Real-Time Conference,
2007 15th IEEE-NPSS, pages 1–5. IEEE, 2007.
126

Bibliography127[10] I Alexandrov, A Amorim, E Badescu, M Barczyk, D Burckhart-Chromek,
M Caprini, J Da Silva Conceicao, J Flammer, B Di Girolamo, M Dobson, et al.
Online software for the atlas test beam data acquisition system. Nuclear Science,
IEEE Transactions on, 51(3):578–584, 2004.
[11] High Energy Accelerator Research Organization. Summary of activities of thephoton factory for the period 2000-2005, 2005. URL http://pfwww.kek.jp/
hyoka05/.
[12] N Yamamoto. Recent activities toward worldwide remote operations of accelerators. Proceedings of ICALEPCS2003 Gyeongju Korea, pages 603–605, 2003. URL
http://accelconf.web.cern.ch/Accelconf/ica03/PAPERS/FR101.PDF.
[13] H. Sawa. Collaboratory in photon factory. In International Symposium on Frontierin Materials Design, Synthesis and Measurements, Hyogo, JAPAN, May 2005.
[14] Yukito Furukawa, Kazuya Hasegawa, Daisuke Maeda, and Go Ueno. Developmentof remote experiment system. Proceedings of ICALEPCS2009 Kobe Japan, page615, 2009.

URL http://accelconf.web.cern.ch/AccelConf/icalepcs2009/

papers/wed002.pdf.
[15] Yukito Furukawa, Kazuya Hasegawa, and Go Ueno.

First operation of thewide-area remote experiment system. Proceedings of ICALEPCS2011 Grenoble France, pages 1193–1195, 2011.

URL http://accelconf.web.cern.ch/accelconf/icalepcs2011/papers/thbhaust05.pdf.
[16] Skype web page, 2014. URL http://skype.com.
[17] Facebook web page, 2014. URL http://facebook.com.
[18] Google web page, 2014. URL http://google.com.
[19] A Akiyama, K Ishii, E Kadokura, T Katoh, E Kikutani, Y Kimura, I Komada,
K Kudo, S Kurokawa, K Oide, et al. Computer control system of tristan. Nuclear
Science, IEEE Transactions on, 28(3):2359–2361, 1984.
[20] Takeshi Wada, Tadashi Kambara, Ichiro Yokoyama, Kazuo Shimizu, Hideki
Takebe, Makoto Nagase, and Hiromichi Kamitsubo. Control system of riken heavy
ion accelerator complex. Japanese Journal of Applied Physics, 30(part 1):2947–
2955, 1991.
[21] RE Mielen. Design and performance of the stanford linear collider control system.
Nuclear Science, IEEE Transactions on, 32(1):230–237, 1985.

Bibliography128[22] Isamu Abe, Hitoshi Kobayashi, and Masahiko Tanaka. Pc based control systemusing activex in the kek e-/e linac. Proceedings of PCaPAC99, Tsukuba, Japan.
[23] Leo R Dalesio, Jeffrey O Hill, Martin Kraimer, Stephen Lewis, Douglas Murray,
Stephan Hunt, William Watson, Matthias Clausen, and John Dalesio. The experimental physics and industrial control system architecture: past, present, andfuture. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 352(1):179–184, 1994.
[24] Piotr Pucyk, Tomasz Jezynski, Waldemar Koprek, Tomasz Czarski, Krzysztof T
Pozniak, and Ryszard S Romaniuk. Doocs server and client application for fpgabased tesla cavity controller and simulator. In Wilga-DL Tentative, pages 52–60.
International Society for Optics and Photonics, 2005.
[25] R Tanaka, T Fukui, K Kobayashi, T Masuda, A Taketani, T Wada, and A Yamashita. The first operation of control system at the spring-8 storage ring. Proceedings of ICALEPCS97, 1997.
[26] Takahiro Matsumoto, Yukito Furukawa, and Miho Ishii. Development of newcontrol framework madoca ii at spring-8. In Proceedings of Particle Accelerator
Society Meeting 2013, Nagoya, Japan, 2013.
[27] JM Chaize, A Götz, WD Klotz, J Meyer, M Perez, E Taurel, and P Verdier. The
esrf tango control system status. 8th International Conference on Accelerator &Large Experimental Physics Control Systems, 2001, San Jose, California, 2001.
[28] P Betinelli-Deck, A Buteau, D Corruble, B Gagey, N Leclercq, M Ounsy, andJP Ricaud. Status of the soleil control system. 10th ICALEPCS Int. Conf. on
Accelerator & Large Expt. Physics Control Systems. Geneva, 2005.
[29] JM Chaize. Tango control system status, 2006. URL http://www.jlab.org/
conferences/PCaPAC/talks/Chaize.pdf.
[30] A Uchiyama, H Yamauchi, J Odagiri, M Komiyama, and M Fujimaki. Implementation of pic-based embedded i/o controller with epics for rilac control system.
Proceedings of ICALEPCS2009 Kobe Japan, .
[31] J Odagiri, J Chiba, K Furukawa, N Kamikubota, T Katoh, H Nakagawa, N Yamamoto, M Komiyama, I Yokoyama, H Song, et al. Epics device/driver support
modules for network-based intelligent controllers. Proceedings of ICALEPCS2003,
Gyeongju, Korea, 2003.
[32] Martin R Kraimer, Mark Rivers, and Eric Norum. Epics: Asynchronous driver
support. ICALEPCS05, Geneva, 2005.

Bibliography129[33] Toshiya Tanabe, Toshikatsu Masuoka, Koichi Yoshida, Keiko Kumagai, Misaki
Komiyama, and Takashi Emoto. Current status of the control system development
at riken ri-beam factory. Proceedings of ICALEPCS2003, Gyeongju, Korea, pages
597–599, 2003.
[34] S Michizono, S Anami, M Kawamura, S Yamaguchi, and T Kobayashi. Digital rf
control system for 400—mev proton linac of jaeri/kek joint project. Proceedingsof the 2002 Linac Conference, Gyeongju, Korea, 2002.
[35] S Michizono, S Anami, S Yamaguchi, and T Kobayashi. Digital feedback system
for j-parc linac rf source. Proceedings of the 22nd International Linear Accelerator
Conference, Luebeck, Germany, 2004.
[36] M Kwon, IS Choi, JW Choi, JS Hong, MC Keum, KH Kim, MG Kim, MK Park,SH Seo, S Baek, et al. The control system of kstar. Fusion engineering and design,
71(1):17–21, 2004.
[37] J Odagiri, S Araki, K Furukawa, N Kamikubota, A Kiyomichi, H Nakagawa, andTT Nakamura. Application of epics on f3rp61 to accelerator control. Proceedingsof ICALEPCS2009, Kobe, Japan, pages 916–918, 2009.
[38] A Uchiyama, K Furukawa, N Kamikubota, H Nakagawa, TT Nakamura, J Odagiri,
M Tomizawa, N Yamamoto, K Kameda, T Natsui, et al. Development of embedded
epics on f3rp61-2l. Proceedings of PCaPAC08, Ljubljana, Slovenia, page 145, 2008.
[39] Takuya Nakamura, Kazuro Furukawa, Tatsuro Nakamura, and Jun-Ichi Odagiri.
Upgrading the control system of the movable masks for kekb. In Proceedingsof Particle Accelerator Society Meeting 2009, JAEA, Tokai, Naka-gun, Ibaraki,
Japan, .
[40] M Takagi, N Kamikubota, A Kiyomichi, S Murasugi, R Muto, H Nakagawa, J Odagiri, K Okamura, and N Nagura. Control of the j-parc slow extraction line basedon embedded epics. Proceedings of ICALEPCS2009, Kobe, Japan, pages 549–551,
2009.
[41] M Komiyama, M Fujimaki, N Fukunishi, RIKEN Nishina Center, A Uchiyama,
and J Odagiri. Upgrading the control system of riken ri beam factory for new
injector. Proceedings of ICALEPCS2009, Kobe, Japan, pages 275–277, 2009.
[42] Jian Zhuang, Kejun Zhu, Yuanping Chu, Jiajie Li, Lei Hu, and Dapeng Jin. The
performance test of f3rp61 and its applications in csns experimental control system.
Proceedings of ICALEPCS2011, Grenoble, France, pages 763–766, 2011.

Bibliography130[43] CY Wu, CS Fann, Jenny Chen, YS Cheng, SY Hsu, Demi Lee, KH Hu, CH Kuo,
KT Hsu, KK Lin, et al. Control of the pulse magnet power supply by epics ioc
embedded plc. Proceedings of IPAC’10, Kyoto, Japan, pages 2731–2733, 2010.
[44] A. UCHIYAMA, M. Kobayashi-Komiyama, E. Ikezawa, M. Fujimaki, T. Ohki,
H. Yamauchi, and N. Fukunishi. Replacement of aging controller gmacs with
f3rp61-2l for power supplies of quadrupole magnets in rilac. RIKEN Accelerator
Progress Report, 45, 2012.
[45] K Rehlich.

Status of the control system for the european xfel.

Proceed-ings of ICALEPCS2011, Grenoble, France, pages 597–599, 2011. URL http://accelconf.web.cern.ch/Accelconf/icalepcs2011/papers/tudaust04.pdf.
[46] J Odagiri, K Akai, K Furukawa, S Michizono, T Miura, TT Nakamura, H Deguchi,
K Hayashi, J Mizuno, and M Ryoshi. Fully embedded epics-based control of lowlevel rf system for superkekb. Proceedings of IPAC’10, Kyoto, Japan, pages 2686–
2688, 2010.
[47] K Furukawa, K Akai, A Akiyama, T Kobayashi, S Michizono, T Miura, K Nakanishi, H Deguchi, K Hayashi, and M Ryoshi. Embedded llrf controller with channel access on microtca backplane interconnect. Proceedings of ICALEPCS2011,
Grenoble, France, pages 1274–1276, 2011.
[48] Leo R Dalesio, MR Kraimer, and AJ Kozubal. Epics architecture. In ICALEPCS,
volume 91, pages 92–15, 1991.
[49] Epics-base r3.14.12.1 source code.

URL http://www.aps.anl.gov/epics/

download/base/index.php.
[50] Martin R. Kraimer, Janet B. Anderson, Andrew N. Johnson, W. Eric Norum,
Jeffrey O. Hill, Ralph Lange, Benjamin Franksen, and Peter Denison. Epics application developer’s guide. URL http://www.aps.anl.gov/epics/base/R3-14/
11-docs/AppDevGuide.pdf.
[51] Kenneth Evans Jr. An overview of medm. In Proc. ICALEPCS 1999 Conference,
Trieste, Italy, pages 466–468, 1999.
[52] R Keitel.

Edlbuild–display generation for the epics edm display manager’.

ICALEPCS05, Geneva, 2005.
[53] Jan Hatje, M Clausen, Ch Gerke, M Moeller, and H Rickens. Control system
studio (css). Proceedings of ICALEPCS07, Knoxville, Tennessee, USA, 2007.

Bibliography131[54] Xihui Chen and Kay Kasemir. Boy, a modern graphical operator interface editor
and runtime. Proceedings of 2011 Particle Accelerator Conference, New York, NY,
USA, 2011.
[55] Kay-Uwe Kasemir and LR Dalesio. Overview of the experimental physics andindustrial control system (epics) channel archiver. Proceedings of 8th International
Conference on Accelerator & Large Experimental Physics Control Systems, 2001,
San Jose, California, 2001. URL http://accelconf.web.cern.ch/AccelConf/
ica01/papers/THAP019.pdf.
[56] Misaki KOMIYAMA, Nobuhisa FUKUNISHI, and Akito UCHIYAMA.

Con-

struction of new data archive system in riken ri beam factory. Proceedings of
ICALEPCS2011, Grenoble, France, pages 90–93, 2011.
[57] TT Nakamura, N Yamamoto, and T Katoh. Data archiving system in kekb accelerators control system. Proceedings of the 10th ICALEPCS, Geneva, 2005.
[58] Akinobu Yoshii and Nobuhiro Kikuzawa. Examination of applying hadoop forj-parc driving data archive. In Proceedings of the 8th Annual Meeting of Particle
Accelerator Society of Japan.
[59] Takuya Nakamura, Kazuro Furukawa, Takashi Obina, and Kay Kasemir. Upgradeof alarm system at pf-ar accelerator control. In Proceedings of the 9th Annual
Meeting of Particle Accelerator Society of Japan, . URL http://www.pasj.jp/
web_publish/pasj9/proceedings/PDF/WEPS/WEPS108.pdf.
[60] F Momal and C Pinto-Pereira. Using world-wide-web for control systems. In
Proceedings 1995 International Conference on Accelerator and Large Experimental
Physics Control Systems, Chicago, IL, Oct, volume 3, 1995. URL http://www-bd.
fnal.gov/icalepcs/abstracts/PDF/m3ac.pdf.
[61] Takuya Nakamura, Kenji Yoshii, Tomohiro Aoyama, Kazuro Furukawa, Tatsuro
Nakamura, and Takashi Obina. Delivery of accelerator operation panel active
images and video image using web server. In Proceedings of the 5th Annual Meetingof Particle Accelerator Society of Japan Hiroshima, 2008. URL http://www.pasj.
jp/web_publish/pasj5_lam33/contents/PDF/TP/TP011.pdf.
[62] T Hirono, T Matsushita, T Ohata, and A Yamashita. Development of data logging
and display system, mydaq2.
[63] Kenzi Yoshii, T Nakamura, K Furukawa, TT Nakamura, T Obina, M Satoh, andN Yamamoto. Web-based electronic operation log system-zlog system. Proceedingsof ICALEPCS07 Knoxville Tennessee USA, 2007. URL https://accelconf.
web.cern.ch/accelconf/ica07/PAPERS/WOAB04.PDF.

Bibliography132[64] Akito Uchiyama, Kazuro Furukawa, Yoshihide Higurashi, Misaki Komiyama,
Takahide Nakagawa, and Kazutaka Ozeki. Design of web-based interface to riken
28 ghz super-conducting ecr ion source and the future plan (to be published).
Proceedings of ECRIS2012 Sydney Australia, 2012.
[65] Takuya Kudou, Shiro Kusano, Kazuro Furukawa, and Masanori Satoh. Upgradeof electronic logbook system at kek injector linac. In Proceedings of the 8th Annual
Meeting of Particle Accelerator Society of Japan Tsukuba, 2011. URL http://
www.pasj.jp/web_publish/pasj8/proceedings/poster/MOPS094.pdf.
[66] N Kamikubota, N Yamamoto, H Nakagawa, S Yamada, KC Sato, T Katoh, J Odagiri, N Kikuzawa, Y Kato, M Kawase, et al. J-parc control toward future reliable
operation. Proceedings of ICALEPCS2011 Grenoble France, pages 378–381, 2011.
URL http://epaper.kek.jp/icalepcs2011/papers/mopms026.pdf.
[67] Kazuro Furukawa, Masanori Satoh, Igor Mejuev, and Keisuke Nakao. A javabased epics archive viewer with soap interface for data retrieval.
ings of ICALEPCS2003 Gyeongju Korea, pages 223–225, 2003.

Proceed-URL http://accelconf.web.cern.ch/AccelConf/ica03/PAPERS/MP707.PDF.
[68] D Quock, N Arnold, and A Johnson. Infrastructure monitoring system for the advanced photon source control system. Proceedings of PCaPAC08 Ljubljana Slovenia, pages 19–21, 2008.

URL https://accelconf.web.cern.ch/AccelConf/

pc08/papers/moz01.pdf.
[69] Thomas Pelaia II and Matthew Boyes.

Introducing caml ii.

Proceedings of

ICALEPCS2009 Kobe Japan, pages 922–924, 2009. URL http://accelconf.
web.cern.ch/AccelConf/icalepcs2009/papers/fra001.pdf.
[70] Lei DUAN and Liren SHEN. Web services interface to epics channel access. Nuclear
Science and Techniques, 19(2):74–78, 2008.
[71] LF Li and CH Wang. A web based realtime monitor on epics data. Proceedings of ICALEPCS2011 Grenoble France, pages 121–123, 2011. URL https://accelconf.web.cern.ch/accelconf/icalepcs2011/papers/mopkn014.pdf.
[72] Kay-Uwe Kasemir, Xihui Chen, John Purcell, and Katia Danilova. Sns online
display technologies for epics. Proceedings of ICALEPCS2011 Grenoble France,
pages 1178–1181, 2011.

URL http://accelconf.web.cern.ch/AccelConf/

icalepcs2011/papers/thbhaust01.pdf.
[73] Y Furukawa. Web-based control application using websocket. Proceedings of
ICALEPCS2011, Grenoble, France, pages 673–675, 2011.

Bibliography133[74] Ian Fette and Alexey Melnikov. The websocket protocol, 2011. URL http://
tools.ietf.org/html/rfc6455.
[75] Microsoft web page, 2012. URL http://msdn.microsoft.com/ja-jp/library/ie/hh673567%28v=vs.85%29.aspx.
[76] A Uchiyama, K Furukawa, and Y Higurashi. Epics channel access using websocket.
Proceedings of PCaPAC2012, Kolkata, India, . URL https://accelconf.web.
cern.ch/accelconf/pcapac2012/papers/wecc02.pdf.
[77] URL http://libwebsockets.org/.
[78] URL http://www.eclipse.org/jetty/.
[79] URL http://nodejs.org.
[80] URL http://socket.io/.
[81] URL http://code.google.com/p/v8/.
[82] Document object model (dom), 2009. URL http://www.w3.org/DOM/.
[83] URL http://www.flotcharts.org/.
[84] URL http://code.google.com/p/jsgauge/.
[85] Takuya Kudou, Shiro Kusano, Masanori Satoh, and Kazuro Furukawa. Development of operation information display panel using websocket at kek injector linac.
In Proceedings of the 10th Annual Meeting of Particle Accelerator Society of Japan,
Nagoya, 2013.
[86] Akito UCHIYAMA, Kazuro Furukawa, and Yoshihide Higurashi. Operator intervention system for remote accelerator diagnostics and support. Proceedings of
ICALEPCS2013, San Francisco, CA, USA. URL http://accelconf.web.cern.
ch/AccelConf/ICALEPCS2013/papers/tuppc110.pdf.
[87] Kenneth Evans and Martin Smith.

Experience with the epics pv gateway

at the aps. In Particle Accelerator Conference, 2005. PAC 2005. Proceedingsof the, pages 3621–3623. IEEE, 2005. URL http://www.aps.anl.gov/epics/
EpicsDocumentation/ExtensionsManuals/Gateway/FPAT064.pdf.
[88] N Kamikubota, S Yamada, N Yamamoto, T Iitsuka, S Motohashi, M Takagi,
S Yoshida, and H Nemoto. Virtual io controllers at j-parc mr using xen. Proc. of
ICALEPCS11, Grenoble, France, 2011. URL http://accelconf.web.cern.ch/
Accelconf/icalepcs2011/papers/wepmu039.pdf.

Bibliography134[89] G Liu, C Li, K Xuan, J Wang, and X Bao. First experience with vmware servers
at hls. Proc. of ICALEPCS11, Grenoble, France, 2011. URL http://accelconf.
web.cern.ch/accelconf/icalepcs2011/papers/mopms004.pdf.
[90] O Khalid and A Shaikh. Optimizing infrastructure for software testing using
virtualization. Proc. of ICALEPCS11, Grenoble, France, 2011. URL http://
accelconf.web.cern.ch/Accelconf/icalepcs2011/papers/webhaust02.pdf.
[91] E Matias, D Beauregard, R Berg, G Black, MJ Boots, W Dolton, D Hunter,
R Igarashi, D Liu D Maxwell, CD Miller, et al. Phase ii and iii the next generationof cls beamline control and data acquisition systems.
[92] Sangil Lee, Jinseop Park, Jaesic Hong, Mikyung Park, and Sangwon Yun. Management tools for distributed control system in kstar. Proc. of ICALEPCS11,
Grenoble, France, 2011.

URL https://accelconf.web.cern.ch/accelconf/

icalepcs2011/papers/wemmu006.pdf.
[93] Akito Uchiyama, Misaki Komiyama, and Nobuhisa Fukunishi. System design
and implementation using virtualization technology for ribf control system. In
Proceedings of the 10th Annual Meeting of Particle Accelerator Society of Japan,
Nagoya, 2013.
[94] Tatsuaki Sakamoto and Takashi Sugimoto. Upgrade of wide area remote control
system (warcs) at spring-8 and sacla. In Proceedings of the 10th Annual Meetingof Particle Accelerator Society of Japan, Nagoya, 2013.
[95] A Yamashita and Y Furukawa. Warcs: Wide area remote control system in spring8. Proc. of 10th ICALEPCS Int. Conf. on Accelerator & Large Expt. Physics Control Systems, 2005. URL http://accelconf.web.cern.ch/accelconf/ica05/
proceedings/pdf/P1_068.pdf.
[96] Yasushige Yano. The riken ri beam factory project: A status report. Nuclear
Instruments and Methods in Physics Research Section B: Beam Interactions with
Materials and Atoms, 261(1):1009–1013, 2007.
[97] Yasushige Yano, A Goto, M Kase, and T Katayama. Ri beam factory project atriken. In AIP CONFERENCE PROCEEDINGS, pages 161–166. IOP INSTITUTE
OF PHYSICS PUBLISHING LTD, 2001. URL http://accelconf.web.cern.ch/
AccelConf/c01/cyc2001/paper/C-1.pdf.
[98] H Okuno, N Fukunishi, and O Kamigaito. Progress of ribf accelerators. Progressof Theoretical and Experimental Physics, 2012(1):03C002, 2012.

Bibliography

135[99] Yoshihide Higurashi. Doctoral thesis in graduate school of science, rikkyo university, 2001.
[100] T Nakagawa, Y Higurashi, J Ohnishi, T Aihara, M Tamura, A Uchiyama,
H Okuno, K Kusaka, M Kidera, E Ikezawa, et al. First results from the newriken superconducting electron cyclotron resonance ion source (invited). Review
of Scientific Instruments, 81(2):02A320, 2010.
[101] Y Higurashi, J Ohnishi, T Nakagawa, H Haba, M Tamura, T Aihara, M Fujimaki,
M Komiyama, A Uchiyama, and O Kamigaito. Results of riken superconducting
electron cyclotron resonance ion source with 28 ghz. Review of Scientific Instruments, 83(2):02A308, 2012.
[102] M Komiyama, M Fujimaki, M Kase, and A Uchiyama.
system for riken ri-beam factory.

Status of control

In Proceedings of the 2007 International

Conference on Accelerator and Large Experimental Physics Control Systems(ICALEPCS2007), Knoxville, Tennessee, USA, 2007. URL http://accelconf.
web.cern.ch/AccelConf/ica07/PAPERS/TPPB11.PDF.
[103] A Uchiyama, R Koyama, M Komiyama, M Fujimaki, and N Fukunishi. Network security system and method for ribf control system.

Proceedings of

ICALEPCS2011, Grenoble, France, . URL http://accelconf.web.cern.ch/
AccelConf/icalepcs2011/papers/wepmu038.pdf.
[104] A Uchiyama, K Furukawa, Y Higurashi, K Ozeki, M Komiyama, and T Nakagawa.
Design of web-based interface to riken 28 ghz super-conducting ecr ion source andthe future plan. Proceedings of ECRIS2012, Sydney, Australia, . URL http://accelconf.web.cern.ch/AccelConf/ecris2012/papers/tupp12.pdf.
[105] A Uchiyama, K Furukawa, Y Higurashi, and T Nakagawa. Implementation of an
operator intervention system for remote control of the riken 28 ghz superconducting electron cyclotron resonance ion source. Review of Scientific Instruments, 85(2):02A904, 2013.
[106] Akamai Technologies. The state of the internet. Akamai Report, 6(1), 2013. URL
http://www.akamai.com/stateoftheinternet/.
[107] A Žagar, K Žagar, K Furukawa, and R Rechenmacher.

Network analyser

for the epics channel access protocol. URL http://accelconf.web.cern.ch/accelconf/pc08/papers/tup008.pdf.
[108] Webpage of apposite technologies, inc., 2014. URL http://www.apposite-tech.
com/products/mini2.html.
[109] Jubatus web page, 2014. URL http://jubat.us/ja/.