OPERATOR INTERVENTION SYSTEM FOR REMOTE ACCELERATORDIAGNOSTICS AND SUPPORT
A. Uchiyama#, The Graduate University for Advanced Studies (SOKENDAI), Tsukuba, JapanK. Furukawa, High Energy Accelerator Research Organization (KEK), Tsukuba, Japan
Y. Higurashi, RIKEN Nishina Center, Wako, Japan
Abstract
Large experimental physics projects, such as ITER andLHC, are typically managed by international
collaboration. Similarly, the International Linear Collider(ILC), a next generation project, will be launched as aresult of the collaborative efforts of multiple institutes
fromthreeregions.
Afterits
collaborativecommencement, all collaborators apart from the hostcountry will need to have methods for remote
maintenance, control, and monitoring of its associateddevices. For example, a method has to be provided forconnecting to the control system network via wide area
network (WAN) links from various collaborating
institutions. However, from a practical application
standpoint, the remote operation of an accelerator via
WAN is beset by a number of issues. One such issue isthat the accelerator has both experimental device andradiation generator characteristics. Additionally, any
mistake operation of the remote control system could
result in an immediate breakdown. For this reason, we
propose the implementation of an operator intervening
system for remote accelerator diagnostics and support thatcan obviate any differences between the local control
room and remote locations.

INTRODUCTION
In an international collaborative project such as theInternational Linear Collider (ILC), remote operation isrequired in order for collaborators to utilize, monitor, andcontrol the facility in the host country. Therefore,
implementation of the global accelerator network (GAN)
as a dedicated network for the ILC control system has
been planned since March 2000 [1]; however, it remained
unrealized until now. The idea underlying GAN is toconstruct remote accelerator control rooms that arevirtually equivalent in terms of performance and features
to the local control room in the ILC, for use by remoteusers of the facility. To realize the GAN project, it isnecessary to implement a remote control systemcomprising an operator interface (OPI), a videoconference
system, an electronic log system, and an online fault
diagnosis system for accelerator devices. In addition,
because communication between on-site operators andremote users is essential, all information about activities
taking place in the local control room should becommunicated to the remote operators.

______________________________________________

a-uchi@riken.jp#A similar system to GAN, called the Global
Accelerator Network Multipurpose Virtual Laboratory(GANMVL), was implemented in the EUROTeV project[2]. GANMVL facilitates remote operation using a clientsystem that utilizes X11 windowing, virtual network
computing (VNC), and JAVA VNC. However, thesecommunication methods encounter issues such as lowbandwidth and high network latency during remote
operation over wide area links. Thus, with the limited
network resources present in some areas, if these methods
are used, it may prove difficult to stably operate andcontrol the accelerator. Additionally, in GANMVL, thevideo/audio communication tools are invaluable inallowing experts to work together at the global level on
accelerator operation problems. This is because they help
to mitigate communication apprehension caused bydifferences in the native languages of the local staff andsome of the global experts.
Their situation is similar to ours as regards thecooperation of experts and remote operation across WAN
links. In situations such as accelerator troubles andchange of beam conditions (beam emittance from ion
sources, etc.), on-site accelerator operators need to seek
direction from other engineers or scientists about
accelerator components, for example, radio frequency(RF), vacuum, beam diagnostic, control system, and ion
source, over the telephone. In such scenarios, the ability
to give the exact directions required by accelerator
operators is contingent on the amount of informationneeded for remote troubleshooting. To prevent
operational mistakes, the accelerator conditions need to beunderstood as a whole. Even in cases where the native
languages are not in conflict, it is difficult to correctly
provide detailed information about accelerator conditions
via telephone and voice communication because too little
information is obtained by listening compared withseeing. Moreover, the GANMVL project reported that itwas not necessary to consider a safety system in themonitoring of their system via remote access fromoutside. Note that a safety mechanism is indispensable
when the remote operation needs not only to monitor theparameters but also to output control in order to prevent
accelerator trouble caused by operational mistakes. To
address safety issues and improve usability for remoteoperations, we developed an operator intervention system
for WebSocket [3] access in the Experimental Physics andIndustrial Control System (EPICS).

WEBSOCKET-BASED OPIWebSocket was originally designed to be implemented aspart of HTML5 and was standardized as RFC6455 by theIETF in 2011 [3]. WebSocket realizes a reasonableinteractive response using bidirectional communicationand thereby overcome the disadvantage that could not be
eliminated using traditional asynchronous JavaScript andXML (Ajax) web applications. Furthermore, sinceperiodic polling is not necessary like Ajax, it becomes
possible to reduce network traffic. Major web browsers
such as Firefox, Chrome, Safari, and Internet Explorer
can be used as cross-platform environments with theimplementation of a WebSocket-based OPI (see Fig. 1).
This allows the user to access the OPI from both a PCbased web browser and other devices such as Android andApple smartphones and tablets. A prototype WebSocketbased OPI was constructed for the message and databaseoriented control architecture (MADOCA) developed atSpring-8 [4]. We also developed a WebSocket-based OPIfor EPICS [5]. After satisfying the conditions forinteractive access to EPICS, we implemented aWebSocket server, which connects to the EPICS IOC via
Channel Access (CA), as a web-based OPI. The
WebSocket server gets/puts the values as process
variables (PVs) from the EPICS IOC by calling the CA
API. Therefore, if security is not taken into consideration,
it is technically possible to access and control the systemusing the WebSocket connections from outside theaccelerator control network, such as from the Internet.

OPERATOR INTERVENTION SYSTEM
System Concept
The purpose of the operator intervention system is toensure safe remote output control of EPICS via theWebSocket server. To simply monitor the status of theaccelerator parameters using the WebSocket-based OPI,
permission from the on-site operator is not necessary as
long as authentication has been accomplished with an
SSL connection. The WebSocket-based client system isdesigned so that output control via remote operation
always requires the permission of an on-site accelerator
operator who can intervene at any time. The main part ofthe operator intervention system consists of a PV gateway
provided by the EPICS collaboration [6], a MySQL
database, and Ajax web applications. Unlike in the OPI,
real-time interactive response is not required for sending
output control permission and to view access logs in theoperator intervention system. Consequently, in
consideration of the difficulty of system development, weutilized a LAMP (Linux, Apache, MySQL, PHP)
environment for the operator intervention system without
using WebSocket.
The on-site accelerator operator first gives permission toevery output of the PVs corresponding to the hardware
devices of EPICS. If the PVs have not been granted
permission, then the operator intervention system always
rejects the output command in the CA protocol layer. Inaddition, the on-site operator can decide on upper andlower limits for remote control operations before sending
the permission command.

System Policy

Figure. 1: Screen shot of WebSocket-based
OPI on Apple iPhone4s

The operator intervention system can control theavailability of remote operations by sending system flagvalues to the MySQL database. The access control
mechanism for the CA protocol is utilized by the accesscontrol security system of the PV gateway. Specifically, adaemon polls the database and updates the security accessfile (GATEWAY.pvlist) with information corresponding
to the flag values dynamically. A flowchart illustrating
the operation of the system for remote connections isdepicted in Fig. 2. First, remote users send EPICS PVs
requests to the operator intervention system for control ofthe outputs, such as caput command, via WebSocket (Flag0). The on-site operator then ascertains whether therequested PVs are available for control (Flag 1). If control
is possible, a command to accept is sent by the on-siteoperator (Flag 2); otherwise, a refuse command is sent(Flag 3). The on-site operator can terminate the remote
operation at any time, for example, if problems with theaccelerator arise (Flag 3). A process that manages thetime duration of the remote operation sends a timeout
command after a certain period of time has elapsed (Flag4).

Virtualization Technology
The access control security system does notcomprehensively cover access from the Internet because
EPICS is not able to completely prevent impersonation.
Generally, a defense-in-depth network system, which hasa multi-layer network structure, is one of the most useful
methods for improvement of network security. In oursystem, we implemented a multi-layer network structure
consisting of servers running in a virtual environment on
one physical server. In the system, network attacks andunlawful access are thwarted by using virtual machines toprevent access from the physical network (see Fig. 4). We
implemented the virtual environment using VMware
vSphere Hypervisor 5, which can be used free of cost.
The specifications for the physical server are shown in
Table 1. We used CentOS 5.9 as the operating system forthe virtual machines.

Figure 2: Flowchart illustrating the opration of thesystem for remote.

Required Services
The operator intervention system comprises important
services running on several distributed hosts. A chart ofthe system is depicted in Fig. 3. The MySQL service isnecessary to store the flag value via an internal network.
The Apache web server and NFS run on an application
server. NFS is used to share the log files among theWebSocket server, the Apache web server, and the PV
gateway via the internal network, as well as the MySQL
service. WebSocket server programs, developed using
Node.js [7], run on another host that connects to theMySQL database and the EPICS IOCs via the PV
gateway.

Table 1: Specification for the physical server used in thevirtualization environment.
CPUIntel Core i7-3770(4 cores, 8 threads)
Memory16 GB
Storage

SATA

Ethernet

Dual network connections

Figure 4: Network chart of the system in theVMware virtualized environment

Customized PV Gateway

Figure 3: System chart showing the operator intervention
system, EPICS IOCs, and the WebSocket server.

In general, EPICS utilizes DRVH/DRVL fields to set theupper and lower limits of analog output (ao) records [8].
In this case, however, utilizing the DRVF/DRVL fields ofthe EPICS IOCs to control the system network is not wise
because it may cause problems, such as lack of parameter
resetting, in normal operation with remote users. As aresult, we implemented the function by modifying thesource code of the PV gateway. Consequently, ourcustomized PV gateway has a GATEWAY.limit file to set
the upper and lower limits as well as GATEWAY.pvlist
and GATEWAY.access files for access control.

User Interface

CONCLUSION

We developed the interface of the operator intervention
system as a web application using Ajax because it is not
required to be as responsive as the WebSocket-based OPI.
The user interface of the operator intervention system isshown in Fig. 5. On-site operators can check therequested EPICS PVs, completed PVs, and in-operation
PVs in one view. Similarly, all the access logs and caput
logs are available for checking by the on-site operator at
any time.

Security for WebSocket-side
We ensure secure WebSocket access to the accelerator
network from the WAN by using a virtual private network(VPN) and authentication with SSL. In addition,
accelerator operators have to comprehensively verify auser (via login ID, etc.) seeking access to the networks
and access route (full domain of Internet providers, etc.)
before WebSocket control is allowed.

IMPLEMENTATIONAt the RIKEN Radioactive Isotope Beam Factory(RIBF), we implemented an operator intervention system
for the EPICS-based RIKEN 28 GHz superconfucting
ECR ion source (28GHz SC-ECRIS) control system foran evaluation [9]. Using a WAN emulator producted by
Apposite Technologies, Inc. [10], we emulated Internetaccess for a speed of 2.0 Mbps and a latency of 200 ms.
Consequently, we were able to successfully operate partof the 28 GHz SC-ECRIS (gas valves, RF power, andelectrode positions) remotely without any serious glitches.
The on-site operators managed the digital and analog
outputs for remote operation, and access was denied in thenetwork protocol layer if certain criteria were not met.

In this paper, we proposed an operator intervention
system for a WebSocket-based OPI in the Experimental
Physics and Industrial Control System (EPICS). In orderto prevent different kinds of troubles in far remote
operation, a safety system is necessary. This system
ensures secure remote operations, including for outputdevices, by allowing the intervention of an on-site
operator. Following the implementation of the operator
intervention system, it now facilitates the monitoring of avacuum and a beam current as numerical values in thereal-time web from remote locations. In addition, it makes
it possible to perform output control without any lapse inoperation safety.

REFERENCES[1] “A Global Accelerator Network: ICFA Task Force
Reports”, Dec. 2001;
http://www.fnal.gov/directorate/icfa/icfa_tforce_reports.html[2] R. Pugliese, et al. Proceedings of ICALEPCS07, Knoxville,Tennessee, USA, P.418-P.420[3] I. Fette and A. Melnikov, The WebSocket Protocol, IETF
HyBi Working Group. 2011.
[4] Y. Furukawa, et al. Proceedings of ICALPECS 2011,
Grenoble, France, 2011, WEMAU010.
[5] A. Uchiyama, et al. Proceedings of PCaPAC2012, Kolkata,India, 2012, WECC02[6] K. Evans, et al. Proceedings of 10th ICALPECS, Geneva,
2005[7] http://nodejs.org[8] EPICS Application Developer’s Guide,
http://www.aps.anl.gov/epics/[9] A. Uchiyama, et al. Rev. Sci. Instrum. (to be published)
KEK Preprint 2013-38,
http://www-lib.kek.jp/tiff/2013/1327/1327038.pdf[10] http://www.apposite-tech.com/products/mini2.html

Figure 4: The user interface of the operator intervention system implemented in Ajax.