Improvements to Realize a Higher Reliability of the KEK Linac Control System
Norihiko Kamikubota, Kazuro Furukawa, Kazuo Nakahara, Isamu Abe and Akihiro Shirakawa
National Laboratory for High Energy Physics (KEK)
1-1 Oho, Tsukuba, Ibaraki 305, Japan

Abstract
A new control system for the KEK 2.5-GeV electron/positron linac, which comprises VME stations with the OS-9 operating system and UNIX-based
workstations, started operation in October 1993. During the 10000 hours of operation in the first two years, the failure rate of the control system was lessthan 1%. However, two-year operation has provided several suggestions as to improving the reliability of the control system. On the basis of our current
experience several kinds of improvements have been carried out or are in progress.

I. Introduction
The new control system for the KEK 2.5-GeV electron/positron linac started operating in October 1993 [1], [2]. It comprises
three components:(a) UNIX-based workstations as resource servers,(b) the TCP/IP network as a main communication path, and(c) VME-bus computers with the OS-9 operating system as front-end interfaces.
During the 10000 hours of operation in the first two years, the failure rate of the control system was less than 1%1 .
However, we experienced a few problems which forced us to stop the control system for more than one hour. The mostserious one was due to a disk crash. At the first occurrence it forced us to stop the control system for ten hours. In order torealize a higher reliability of our control system several kinds of improvements have been carried out or are in progress.
In this report, the improvements concerning hardware redundancy are described in Section II. The developments in thecontrol software are discussed in Section III.

II. Hardware redundancy
A. Workstations to distribute functions
When the new system started in October 1993 we had only two UNIX workstations as resource servers.2 One was used
for the operation (hostname peach), and the other was for software development (maple). At that time all of the functions
needed for linac operation were concentrated on peach.
We experienced two problems: One was that the resources of peach, such as the CPU power and the memory, had become
inadequate for the increasing demand. The other was that a problem concerning peach inevitably resulted in a stoppage ofthe linac operation. It was apparent that a backup scheme to cover the fault of peach was desirable. We thus decided tointroduce a new workstation with higher performance as a secondary machine for linac operation.
Since our inter-process communication protocol has availability among workstations of different operating systems [3], wewere able to choose a workstation from among a wide variety of candidates having various operating systems. We selected
the DEC3000AXP, since it seemed to have the best cost-performance at the time. The new workstation, called grape, isroughly 3–5 times as powerful as the previous one, peach. A list of the workstations in our control system is given in Table I.
1The total operation hours in fiscal year 1994 was 5040 hours, while the failure time due to the problems concerning the control system was 13.4 hours.
However, they were not capable of acting as
resource servers.
2 We also had a disk-less UNIX machine, lime, as well as a few old-fashioned UNIX-based machines.

In order to proceed with a further distribution of the functions, we will introduce two additional workstations by the endof fiscal year 1995. One will be used as a secondary machine for software development with the same operating system asgrape; the other will be used as the third operation server.3 Both are also given in Table I.
Table I

List of the UNIX workstations in the KEK linac control system.
hostname
peach
lime
maple
grape
-machine(operating system)
DECstation5000(Ultrix v4.4)
DECstation5000(Ultrix v4.4)
DECstation5000(Ultrix v4.4)
DEC3000AXP(Digital UNIX v3.2)
DEC3000AXP(Digital UNIX)
DEC3000AXP(Digital UNIX)

introduced inNov.1990
Oct.1991
Oct.1992
Mar.1994
Feb.1996 (plan)
Feb.1996 (plan)main function
operation 1status display
development 1
operation 2
development 2
operation 3

B. Disk-array unit
During the years before 1994 we experienced a few problems due to disk-crashes. Fortunately these problems were notserious, since the disks did not contain any important files necessary for linac operation.
In order not to suffer from such problems we introduced a disk-array unit in July, 1994. The unit contains seven disks of 2
GB and works as a single disk with a size of 12 GB. Any fault of one disk does not affect to the disk-array operation at all.4
We now keep any important files in the disk-array unit. It is worth noting that the files for the VME stations are also kept
there.
C. Network improvements
We started a new control system with a few network segments. In October 1993 we used the popular Ethernet (coaxialcables called 10Base5), linked together with a network repeater. Each VME station in the five sub-control rooms is locatedat 80-meter intervals, and is connected by a long coaxial cable. Since there were several high-power klystrons along thecable, we were afraid of some influence of pulsed noise from them.
The first improvement was carried out in July 1994. We introduced fiber-optic cables between the center room and thesub-control rooms, which are used instead of the coaxial cable. In addition a star-topology is preferable, since any problems
at a sub-control room do not affect the other rooms. Each of the coaxial cables and the fiber-optic network has a capabilityof 10 Mbps throughput.
Since a natural increase of the network traffic was expected, a network hub, having a backbone capable of 100 Mbps
communication throughput, was introduced in March 1995. All of the network segments were connected with the hub, andeach segment was separated by a network bridge. In addition, in order to separate the network traffic concerning only theworkstations in the center room, a FDDI link between the major workstations was introduced in June 1995.
D. Other notes
Since the VME station called kannaduki was in charge of both the injector and the center parts, a new VME station, hatsuhi,
was introduced in September 1994 to accept the functions for the center part.
We sometimes experience a momentary power outage caused by lightning during the summer. In order to avoid such problems
a couple of UPS systems (Uninterruptible Power Source) were introduced in January 1995. They cover the power supplies
at two workstations for linac operation (peach and grape).
3 See also the descriptions in III-A.
4 During the recent 1 year, the disk-array unit unfortunately stopped twice due to unknown errors. We will replace it with a new one to be introduced withthe additional workstations in February 1996 (see Table I).

1st server (peach)

2nd server (grape)application
application
process
process

application
application
process
process

message
distributormessagedistributor

server server server
VME#1 VME#2 VME#3

server
VME#nFigure. 1. Relation between the two servers and the VME stations.

III. Software techniques
A. Two servers for the linac operation
As described in II-A, a new workstation, grape, was introduced as a secondary server for the linac operation in March 1994.
The relation between the two servers and the VME stations is shown in Figure 1. The great advantage of this scheme is thatwhen one of the two servers stops due either to a problem or to a maintenance shutdown, operators are able to continue linac
operation with the remainder server.
Since the operating system of the secondary server (Digital UNIX) is different from that of the first server (Ultrix), we have
prepared the server processes and the application processes for the secondary server by re-compiling the sources written in
C language. Each of the control services and associated processes, such as the digital-output and field networks (LOOP-IIor LOOP-III), has been prepared one by one, as given in Table II.
Table II

Schedule of the control services to arrange for the secondary server, grape.
service nameout16in16
dac12adc12
loop3
loop2
gpib
rs232cdescription
16bit digital output
16bit digital input
12bit analog output
12bit analog input
48kbps field network
500kbps field network
for GPIB instruments
serial line

completed inDec.1994
Feb.1995
Feb.1995
Feb.1995
Jun.1995
Dec.1995(plan)
Feb.1996(plan)
Mar.1996(plan)

As shown in Table I, we will introduce a third server with the operating system Digital UNIX. In order to decrease maintenancework involving the two operating systems, the first server, peach, will be replaced by it.
B. Use of the UDP protocol instead of the TCP protocol
The inter-process communication for our control system has relied on a stream socket based on the TCP protocol [3]. Since
the TCP protocol contains an automatic error-retry scheme, we always wait for roughly one minute to receive a time-out
signal. This is too long of a duration for typical linac operators; however the TCP protocol does not provide a method tochange the time-out period. We have thus decided to use the UDP protocol instead of the TCP protocol.5
5 We use the UDP protocol for the basic services given in Table II. For other services, we continue to use the TCP protocol, since most of the communications are made between two processes in one workstation; that is, no time-out errors are expected.

We also developed a basic inter-process communication procedure with the UDP protocol in August 1994. The procedureincludes three retries with a 2-second time-out interval (8-second time- out in total). The modifications to switch over thecontrol services to use the UDP protocol have been carried out one by one, together with the work involving the secondary
operation server. The present modification is also effective for reducing the overall round-trip time of a control message.
The round-trip time between processes at peach and at a VME station, including overhead with the control system, is about
40 ms with the TCP protocol, while it is 14 ms with the UDP protocol. The round- trip between grape and a VME stationis 8 ms with the UDP protocol.
C. Disk-less VME stations
Each VME station has a local disk which is used for bootstrap loading. Since such a local disk has a greater possibility
to cause problems than the other components in a VME-bus system, studies to realize a disk-less environment were madeduring the summer shutdown of 1995. We changed each VME station so as to use a network bootstrap protocol (BOOT-P),
which enables us to boot from remote files. Since August 1995, the files needed for the bootstrap have been kept on the disksof both the two operation servers (peach and grape). Since they are duplicated, a bootstrap of any of the VME stations ispossible even when one of two servers stops.
D. Other notes
Since there are several workstations, VME stations, and PC’s, it is important to synchronize the internal clocks. The NTP(Network Time Protocol) is utilized for this purpose. It seems to keep the internal clocks of the UNIX workstations within
one second (or better) of accuracy.6 The present control system produces several kinds of operation logs and history files forvarious devices. The total amount of such logs and history files is 5–10 MB per one day. In the case of a problem, they arevery useful, since we can extract any device history of interest. The details will be described elsewhere in the future.

IV. AcknowledgementWe wish to thank Professor I. Sato for his kind encouragement during the present work. We also express our best thanks tothe Linac operators for discussions and various kinds of help.

References[1] N. Kamikubota, K. Furukawa, K. Nakahara and I. Abe, Nucl. Instr. Meth. A352(1994)131[2] N. Kamikubota, K. Furukawa, K. Nakahara and I. Abe, Proc. 1994 International Linac Conference (LINAC94), Tsukuba
Japan, August 1994, p.822; KEK-Preprint 94-73[3] N. Kamikubota, K. Furukawa, K. Nakahara and I. Abe, Proc. Int’l Conf. on Accelerator and Large Experimental Physics
Control Systems (ICALEPCS’91), Tsukuba Japan, November 1991, p.318

6 We currently use the OS-9 of version 2.4 (ISP v1.4), with which the NTP is not available. The next version is already available, and the NTP might beavailable with it.