# Presentation: 518533

## IOC Redundancy:Redundancy Monitor Task 

***EPICS Meeting - Redundancy***
**Argonne, June 16, 2006**
**Matthias Clausen, ****DESY**

## XFEL Controls Demands

**A user facility with high demands for 99.8% efficiency**
- The control system may not fail
- Possible scenarios:
**Hot stand by ( seamless switch of IOC‘s)**
**Cold stand by ( switching ‚on demand‘; i.e. for maintenance)**
- Biggest problem: Find radiation hard electronics for the tunnel
**Distributed redundancy in cold stand by may help**
**Most sensors and actuators contain electronics nowadays.**

- flwejkvnwRÖGK

## Design Goal for redundant IOC’s

**From the preamble of the design specification:**
**... ****last and most importantly one major design goal must be matched:**
***Any redundant implementation must make the system more reliable than the non redundant one. Precaution must be taken especially for the detection of errors which shall initiate the failover. This operation should only be activated if there is no doubt that keeping the actual mastership will definitely cause more damage to the controlled system than an automatic failover.***

- flwejkvnwRÖGK

## Fulfilling XFEL Control Demands

**Redundant Process Controllers will help to provide the required MTBR. The following actions can be performed without interrupting cryogenic operations:**
**XFEL Cryogenic Plant: *****Main objective: Maintenance*****(permanent operation for more than one year)**
- Hardware maintenance
- Software maintenance
- Installing new system-/ application- Software
**XFEL Tunnel Installation: *****Main objective: Survive radiation damage *****(MTBR > 1 month)**
- ... same like cryogenic plant
- Seamless switch over of:
** ****Process Controller (IOC)**
**I/O sensor/ -actor**
**Power supplies**

- flwejkvnwRÖGK

## Principle Layout of the XFEL Cryogenic Control System

- P‘

**Principle Layout of the XFEL Cryogenic Control System**

- P

- T

- P

- T

- PLC

- IOC
- (redundant)

- FEC-IOC
- (redundant)

- Ethernet

- Ethernet

- Field Bus

- Field Bus

**XFEL Tunnel**

**Cryogenic Plant**

- Router

- R-Link

- R-Link

- P

- T

- Field Bus

- Control Backbone

- Gateway

- Office Netw.

**6 X**

**10 X**

- flwejkvnwRÖGK

## Redundancy: New developments

**The implementation of the key redundancy tasks on an IOC has been subcontracted to two companies:**
**Redundancy Monitor Task**
- Supervision of the tasks running on an IOC
- Switching IOC’s in case of serious problems on one IOC
**Continuous Control Executive Task**
- Synchronizing the continuous control processing on two IOC’s
- Permanent monitoring of all changes in record processing on an IOC
**Core and main objective of any failover:**
**The resulting status of an IOC after a failover must be a more stable state than the status before the failover.**
- Diagnostic analysis programs must be activated to ensure this.

- flwejkvnwRÖGK

## Redundancy Monitor Task (RMT)Hardware Layout

- flwejkvnwRÖGK

## Redundancy Monitor Task

**Controlling Primary Redundancy Resources (PRR)**
- Drivers
- Tasks
- Watchdog
- Ethernet ports
**Answering and/ or forwarding XML diagnostic and –management requests**

- flwejkvnwRÖGK

## Redundancy Monitor Task (RMT)Process and Interface Design

- flwejkvnwRÖGK

## Redundancy:Control Executive -> Bob’s talk

**Redundancy:****Control Executive -> Bob’s talk**

- Control
- Update
- Process

- Q
- U
- E

- SNL
- Update
- Process

- Q
- U
- E

- Control
- Exec

- Scan
- Task

- Scan
- Task

- Scan
- Task

**:**
**:**

- SNL
- Prog

- SNL
- Prog

- SNL
- Prog

**:**
**:**

- SNL
- Exec

- Redundancy Monitor
- Process

- I/O
- Driver

- I/O
- Driver

- I/O
- Driver

**LAN**

- later

- flwejkvnwRÖGK

## XFEL Schedule (for Cryogenic Installations)

**XFEL Schedule (for Cryogenic Installations)**

- flwejkvnwRÖGK

## Outlook

**Individual tests by the end of June ‘2006**
**Integrated test in July 2006**
**Intensive testing until end of 2006**
- Redundant hardware
**CompactPCI backplane/ CPU**
**Redundant PS**
**Remote reset for PS**
- Beta testing 2007
**Production for Accelerator Module Test Facility (AMTF)**
- 2009
**SNL Redundancy: **
- Basic functionality implemented by Mike Laznovsky (SLAC)
- Redundancy implementation starting middle 2006

- flwejkvnwRÖGK