# Presentation: 235619

## KEKB Accelerator Control System

**KEKB Controls Group**
**2001-08-29**
**by T. Katoh**

## Control System

**Control Computer System**
**Timing Control System**
**Beam Gate Control System**
**Personnel Safety Control System**
**Communication System**
**Control Room and Operators Consoles**

## Contents

**Construction History of KEKB Accelerator Control System**
**KEKB Control System Design**
**System Hardware Configuration**
**System Software Configuration**
**Central Control Room**

## Construction History

**1994.4****	****Started System Design for KEKB**
**1995.2****	****Controls Systems Evaluation**
**1995.5****	****Committee Recommendation to adopt EPICS**
**1995.8****	****First Version of Computer System Specifications**
**1996.1****	****Started to Dismantle TRISTAN Main-Ring**
**1996.3****	****Final Version of Specifications**
**1996.6****	****Opened the Bids**
**1997.3****	****abco1 and 15 IOCs were Installed**
**1997.4****	****Hardware Specifications were Fixed**
**1997.10****	****BT Lines were Commissioned**
**1998.3****	****The Rest of IOCs were Installed**

## Construction History(contd.)

**1998.4****	****Started Software Installation with Link-Persons and Engineers from the Company**
**1998.12****	****Commissioning of KEKB Accelerators**
**1999.3****	****Commissioning with the BELLE Detector**
**1999.4****	****Started Designing PF-AR Control System**
**2000.10****	****Installation of abco2 and 10 IOCs for PF-AR**
**2001.2****	****Started to Dismantle PF-AR Devices**
**2001.7****	****Started to Install or Re-install Equipment**
**2002.1****	****Commissioning of PF-AR**

## KEKB Control System Design

**System Requirements**
**Constraints**
**Basic Concepts**
**System Architecture**

## System Requirements

**Sources**
**Equipment Groups****	**
**Magnet and Power Supply**
**RF**
**Beam Monitor**
**Vacuum**
**Beam Transport**
**Feedback, Facilities, Physics, etc.**
**Operations Group**
**Accelerator Physicists or Commissioning Group**
**Controls Group**

## System Requirements(contd.)

**Requirements**
**All the data that are possible to take should be taken.**
**All the data that are taken should be saved for later analyses.**
**All the operation should be recorded for later inspection.**
**All the machine parameters and information about the machine components should be stored in the database.**

## System Requirements(contd.)

**Requirements(contd.)**
**The Man-Machine Interface should be Operator-Friendly.**
**The Programming Environment should be Programmer-Friendly.**
**The Overall Response Time to an Operator’s Request should be less than a Second unless the Progress of the Process is Indicated.**

## Constraints

**Use CAMAC as an Equipment Interface**
**Schedule**
**Control Systems Final Design****	****    January, 1995**
**Removal of MR equipment****	****	****    January, 1996**
**Control Systems Installation       September, 1996**
**Completed Hardware Installation****	****       March, 1997**
**KEKB Commissioning****	****	****December, 1998**

## Constraints(contd.)

**Limited Man-Power**
**9 KEKB Controls Group Members**
**3 Physicists and 6 Engineers**
**12 Link-Persons from Other Groups**
**2 from Magnet and Power Supplies**
**2 from RF**
**2 from Vacuum**
**2 from Beam Transport**
**2 from Beam Monitor and Feedback**
**1 from Linac, 1 from Physics**

## Basic Concepts

**Standard Model Architecture**
**International Standards**
**CAMAC, VME, VXI, GPIB, FDDI, etc.**
**Existing Software Environment**** : EPICS**
**Separate Computer-bus and Field-buses**
**“****Link-person” System**** : Applications**
**“****Out-Sourcing” : 5 People from Companies**

## Standard Model Architecture

**Presentation Layer (Server)**
**Man-Machine Interface(OPI)**
**Logging, Analyses, Alarm Displays, Database, Global Feedbacks**
**Equipment Control Layer (IOCs)**
**Data Acquisition, Sequence Control, Local Feedbacks**
**Device Interface Layer (Field-buses)**
**Hardware Interfaces**

## The most Important Points

**Presentation Layer**
**Software Development Tools**
**User/Programmer Friendly Tools**
**Ergonomic Consoles**
**LCDs, Macintosh’s, PCs, PDPs, Flat Desks**
**Equipment Control Layer**
**VMEbus : High Reliability, Flexibility**
**Device Interface Layer**
**CAMAC : High Reliability, Well-known**

## Presentation Layer

**Operator’s Consoles : X-Terminals**
**Database Management : ORACLE 7**
**Alarm Generation/Recording**
**Data Logging**
**Data Display : CATV Network**
**Simulation : SAD Program**
**High-Speed Network**
**FDDI, Distributed Shared-Memory Network, **
**100 Base/TX and 10 Base/T**
**Gateway to KEK Laboratory Network : acsad**

## Equipment Control Layer

**Provides Standard Interfaces**
**CAMAC Serial Highway Drivers(Hytec)**
**MXI-bus Drivers for VXI Main Frames(HP)**
**ARCNET Drivers, GPIB Drivers(NI), etc.**
**Computers : VMEbus based IOCs**
**FORCE CPU-40(MC68040)**
**FORCE CPU-60(MC68060)**
**FORCE PowerCore6603(PPC 603e)**
**FORCE PowerCore6750(PPC 750)**
**Operating System : VxWorks**

## Device Interface Layer

**Standard Interfaces**
**CAMAC : RF and Vacuum**
**VXI : Beam Position Monitors**
**GPIB : RF, Vacuum, Magnet Readouts, etc.**
**ARCNET : Magnet Power Supplies**
**	****	****TV Signal Switches**
**MODBUS+ : PLCs**
**RS232: Vacuum Measuring Instruments**

## System Architecture(1)

**Functional Configuration**

## System Architecture(2)

**Actual Configuration**

## System Configuration

## Central Control Room

## abco1 Server Workstation

**PA-RISC 7200 Architecture**
**120MHz CPU Clock**
**4 CPUs**
**2GB of Main Memory**
**4GB Hard Disk Drives**
**FDDI Interface**
**20GB RAID Disk**

## acsad Server Workstations

**Compaq Alpha Server**
**4 Alpha CPUs of 440MHz**
**6 Alpha CPUs of 330MHz**
**1 GB Memory**
**50 GB RAID**
**FDDI Network Interface**
**True 64 Unix Operating System**

## Former KEKB Control System

## KEKB/PF-AR Control System

## abco2 Server Workstation

**PA-RISC 8500 Architecture**
**440MHz CPU Clock**
**2 CPUs**
**1GB of Main Memory**
**36GB Hard Disk Drives**
**FDDI Interface**
**140GB RAID Disk**

## IOC Configuration

- From CCR

## VME IOC System

**Subrack**
**Schroff 16 Slots Subrack**
**Power Supply Module**
**259 Watts, 5V 35A, 12V 5A, -12V 2A**
**14 HP Wide, Schroff MPS8-7746**
**System Monitor Module**
**Mitsubishi DRSJ-01**
**Remote System Reset**
**DC Power Lines Monitor**

## VME Modules Used

**CPUs**
**FORCE CPU-40****	****MC68040 33MHz**
**FORCE CPU-60****	****MC68060 66MHz**
**FORCE PowerCore 6603e PPC 603e**
**FORCE PowerCore 6750 PPC-750 266MHz**
**FORCE PowerCore 6750 PPC-750 400MHz**
**CAMAC Serial Highway Driver**
**HYTEC VSD 2992**

## VME Modules Used (contd.)

**GPIB Controller**
**National Instruments GPIB 1014**
**ARCNET Driver**
**Advanet ARCNET-4**
**VME-MXI Driver**
**Hewlett Packard VME-MXI**
**National Instruments VME-MXI II**
**MODBUS+ Interface Controller**

## Number of IOCs and VME Modules

|  | MG | RF | MO | VA | BT | MV | EN | TI | PH | LI | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IOC |   8  |    8  |   20  |   12  |    4  |   20  |   20  |    2  |    1  |    1  |    96  |
| CAMAC |     -  |    8  |     -  |   12  |    2  |     -  |     -  |    2  |     -  |    1  |     25  |
| GPIB |  8  |     -  |     -  |   12  |    6  |   20  |   20  |     -  |     -  |     -  |     66  |
| RS-232C |  1  |     -  |     -  |    12  |     -  |     -  |     -  |     -  |     -  |     -  |     13  |
| ARCNET |   96 |     -  |     -  |     -  |    4  |     -  |     -  |     -  |     -  |     -  |    100  |
| MXI |     -  |     -  |   20  |     -  |    1  |     -  |     -  |     -  |     -  |     -  |     21  |
| Total |  105  |   8  |   20  |   36  |  13 | 20  |   20  |    2  |   0 |    1  |   225  |

## Equipment Connected

**2,517 Magnet Power Supplies**
**176 ARCNET Segments**
**800 Beam Position Monitors**

## EPICS Records on IOCs

**242,597 EPICS Records on 94 IOCs**
**  ****25,147 EPICS Records on IOCMGD06**
**    ****2,788.5 EPICS Records in Average**
**  ****48,149 kB Max. Allocated Memory**
**    ****1,324 kB Min. Allocated Memory**
**  ****12,142 kB Ave. Allocated Memory**

## Hardware Summary

**Use Standard Buses**
**Reliability****	****: VME, Compact PCI, CAMAC**
**Not ISA, EISA, PCI, Proprietary Buses**
**Separate CPU Bus from Field Buses**
**Isolation, Analog Signal Handling**
**Choose Components Carefully**
**Power Supplies**
**Capacitors, Fans, Connectors**
**Use Ergonomic Equipment**

## Control Consoles

## TRISTAN Consoles

## KEKB and PF-AR Consoles

## KEKB Consoles

**Voice Generator using Power Macintosh**
**Multiple Screen Display for Consoles**
**DELL Optiplex NX1**
**Intel Pentium II 400MHz, 1GHz**
**Colorgraphic Evolution 4**
**Macintosh with Multiple Video Cards**
**X-terminal using IBM Network Station**
**Power PC 406(133MHz)**
**48MB Memory**
**Disk-less / Fan-less System**

## Control Consoles

**18” TFT LCDs(1280x1024 pixels)**
**16” TFT LCDs(1280x1024 pixels)**
**14” TFT LCDs(1024x800 pixels)**
**Book-type Personal Computers**
**DELL Optiplex NX1 and IBM Network Station**
**Multi-Screen Display Controllers**
**Wireless Keyboard/Mouse**
**Power Macintosh with 2/4 Video Controllers**
**Six 40” Plasma Displays as TV Monitors**
**Easy to change Configuration**

## Software

## Software

**Two Language Architecture**
**TRISTAN Experiences : NODAL and PCL**
**Interpretive Language for Applications**
**SAD and python**
**Short Turn-around Time**
**Safe Debuging Environment**
**Not Dedicated Application Software Programmer**
**Compiler Language for Low-Level Software**
**C or C++ for Low-Level Software**
**Quick Response Time**
**Runs Fast**
**Realizes Everything You Want**

## Software (contd.)

**Relational Database Software**
**From the Designing Stage**
**Put All the Accelerator Information**
**Cable Connection Lists**
**Equipment Parameters**
**Physical Parameters : Sizes, Weight, ...**
**Calibration Constants, Fitting Curve Parameters, ...**
**Histories**
**Installation Dates, Repair Histories, Costs, ....**
**Property Numbers and more ...**
**Generates EPICS Database Automatically**

## Software (contd.)

**EPICS Software Toolkit**
**medm**
**Generated from Relational Database**
**Ah : Alarm**
**Ar : Archiving**
**SNL : State Notation Language**

## EPICS Software Used

**medm**
**SAD**
**python**

## medm Applications

## python Applications

## SAD Applications

## EPICS Databse Files

## Linac Portable CA Server

**Linac Console**

**Linac Main Controls**

**Linac Sub-Controls**

**Linac Equipment**

**KEKB Equipment**

**KEKB IOCs**

**KEKB CA Clients**

**Linac CA Server**

## Linac Portable Channel Access

## Application Software in Use

**SAD****	****	****	****	****141**
**medm****	****	****	****	****  74**
**python****	****	****	****	****  42**
**Miscellaneous****	****	****    6**
**Total****	****	****	****	****263**

## EPICS Records on IOCs

**242,597 EPICS Records on 94 IOCs**
**  ****25,147 EPICS Records on IOCMGD06**
**    ****2,788.5 EPICS Records in Average**
**  ****48,149 kB Max. Allocated Memory**
**    ****1,324 kB Min. Allocated Memory**
**  ****12,142 kB Ave. Allocated Memory**

## Software Summary

**EPICS R3.13 Toolkit on HP-UX 10.2**
**medm for Man-Machine Interface**
**X-Window Server Software on the Terminals**
**python for Application Programming**
**SAD for Accelerator Simulation / Operation**
**VxWorks on IOCs**
**Tornado Cross Software Development Sys.**
**CAPFAST for EPICS Runtime Database Dev.**
**ORACLE 7 for Relational Database**
**Portable Channel Access Server for Linac Controls**

## VME CPU Board Benchmarks

## CPU Board Benchmark Test

**       ****FORCE Pcore 6604****	****  FORCE CPU-40**
**CPU      Power PC 604e****	**** ****	****    MC 68040**
**Clock****	****	**** 200 MHz****	****	****        25 MHz**
**Memory****	****	****   16 MB****	****	****	****16 MB**
**L2 Cache****	****	**** 512 kB****	****	****	****  --**

## CPU Load Arising from Scanning Database

**	****	****	****	****PCore 6604****	****	****CPU-40**
**1.0 sec****	****	****	****0.60 %****	****	****    5.8 %**
**0.5 sec****	****	****	****1.2   %****	****	****  12    %**
**0.2 sec****	****	****	****3.0   %****	****	****  27    %**
**0.1 sec****	****	****	****6.1   %****	****	****  56    %**

## Transactions Time Required for Channel Access

**	****	****	****	****   ****PCore 6604****	****	****CPU-40**
**ca_search****	****	****   ****953 us****	****       ****1,189 us**
**ca_put****	****	****	****     ****23 us****	****	****  ****109 us**
**ca_get****	****	****	****     ****58 us****	****	****  ****118 us**
**ca_put&ca_get****	****     ****76 us****	****	****  ****246 us**

## VME-MXI Problems

## VME-MXI Problem

**HP VME-MXI Driver Module : ****National Instruments Products**
**FORCE PowerCore 6603 or 6750 CPU Modules : ****Power PC**
**Tandra Universe or Universe II Chips : ****PCI-VME Bus Bridge Chips**
**CPU Module Halts and Never Restarts Responding to the Reset Signal without Power is switched off**

## Improvements in ARCNET

**A HUB Box contains 3 ARCNET HUBs**
**33 HUB Boxes were Installed for 97 ARCNET Lines**
**Decreases 10****2****/Day/Line ARCNET “Reconfiguration” Errors down to ****Zero**

## Configuration of the Test Bench

**CPU module**

**VME-MXI module**

**System Monitor module**

**VXI-MXI Controller**

**MPX module**

**MPX module**

**RF Voltmeter**

**CPU Modules tested**
**  ****FORCE CPU40: MC68040, 33 MHz**
**  ****FORCE CPU64: MC68060, 66 MHz**
**  ****FORCE PowerCore 6603: PPC 603, Universe Chip**
**  ****FORCE PowerCore 6750: PPC 750, 266 MHz, Univ. II**
**  ****FORCE PowerCore 6750: PPC 750, 400 MHz, Univ. IIB**

## Normal and Abnormal Bus Cycles

- Normal Bus Cycle        Abnormal Bus Cycle

## Abnormal Bus Cycle

## Normal and Abnormal DTACK* Signals

**PowerCore 6603, with Universe Chip**

**Normal                                                       Abnormal**

## Abnormal DTACK* Signal

**PowerCore 6750, 266 MHz, with Universe II**

**Normal                                                        Abnormal**

## Realtime Linux

## Linux

**Disk I/O causes ****20 - 30 ms**** of Latency, possibly Latency can be around ****100 ms or more****.**
**Cause:**
**Non-preemptive**** Kernel **
**Scheduling Algorithm**
**Interrupt Disabling**

## Non-Preemptive Kernel

- Interrupt

## Preemptive Kernel

## EPICS iocCore

- Network

**Channel Access**

**Run-time Database**

**Device Access Layer**

**I/O Buses (VME/PCI/ISA)**

## EPICS under L4-Linux

## Linux System Calls

## RT-thread Preempts Linux

## RT-thread Calls Linux

## What Happens if RT-thread Calls Preempted Linux?