4 June 2007

**CMS FPIX Vulnerability Assessment Workshop Review Findings**

30 May 2007 Presentations

Session: Slow Controls and Fast Interlock

Facilitators: M. Chertok, B. Cooper, L. Bagby, R. Schmitt

FPIX Technical contact: C. Newsom, L. Cremaldi

**General comments from facilitators**

These general comments are intended to outline an approach to
monitoring, interlocks, and controls and to serve as a reminder of
issues to be considered as the systems are developed. In no way are they
intended to suggest that issues have been overlooked.

We consider monitoring, alarms, and interlocks to be critical for the
long-term, safe operation of the FPIX system. Hazards which are
potentially specific to FPIX include excessive irradiation, operation at
an improper temperature, major loss of cooling, and exposure to water
vapor with a concentration sufficient to lead to condensation. We assume
that other hazards, such as fire, flooding, leaks of cooling systems
associated with other sub-detectors, magnet quenches, and electrical
power fluctuations or disruptions are more likely to affect multiple
detector sub-systems than to be specific to FPIX. Verification should be
made that a more global response leads to a proper response for FPIX.

Monitoring can serve the dual purposes of providing parameters useful
during data analysis and providing inputs to the alarm and interlock
systems. The alarm system provides a warning that parameters have
deviated significantly from normal; that may allow abnormal parameters
to be corrected before they are sufficiently out-of-range to cause an
interlock trip. Interlocks should place the system into a safe state
without human intervention if parameters are sufficiently out-of-range.

We note that experts could automatically be notified via an auto-dialer
triggered by monitoring, alarm, and interlock systems. That can allow a
faster response to abnormal conditions, particularly if the operations
crew is overwhelmed by multiple fault conditions. The current plan is to
have email automatically sent to all experts whenever there is a failure
at a critical level.

It is useful to flag the first, and perhaps the second, fault condition
of all alarm and interlock systems, thereby allowing the cause of a
problem to be identified more easily.

The flags and times of the flags are automatically archived.

We understand that the ability to change alarm and interlock logic and
set-points will be controlled administratively and, perhaps, by a system
of keys. That should be documented. This is very important. When I
visited CERN Andromachi showed us just such a system. I will get more
details from her during my next visit.

We suggest that interlock logic should be general enough and reviewed
carefully enough to allow all FPIX operations to be conducted without
the need for hardware jumpers, bypass keys, or software alterations. We
recognize that alarm and interlock set-points may need to be different
for initial system check-out and final system operation. It may be
possible to accomplish that, while still ensuring safe operation, by
limiting the range available for each set-point. No hardware jumpers are
possible. All bypasses are handled by the State Machine software.
Special administrative priviledges (in the software) are required to do
this.

The interlock system should be "fail-safe", that is, it should place
FPIX into a safe condition under single-component and plausible multiple
component failures. That may require some degree of redundancy. For
example, if FPIX power is to be removed to achieve a safe condition,
multiple means to ensure power is removed may be needed. There are
multiple shut-down paths. The first one is through the PVSS system
commands. The second is the PLC. The third is through the DSS system,
for certain types of failures. For power failure, there is only the PLC
and DSS. Since the DSS may not be activated for "lesser" crises (local
power failure) so only one path exists in that case.

Full interlock system functionality should be checked by triggering a
fault condition for each interlock input and appropriate combinations of
inputs and verifying that the response is appropriate. That check should
be made before initial FPIX operation and at periodic intervals
thereafter. If the interlock system is serviced or modified, portions of
the system that could be affected should be checked. Initial testing is
foreseen. Additional testing is clearly needed if we change anything.

As outlined in the document on DCS commissioning plans, there are still
many steps that need completion. Clearly, integration with the CMS-wide
DCS will be a critical step. In a few places, more hardware
(electronics, etc) is needed. Is there a foreseen issue with procuring
any of the necessary parts? No. We have been following this closely. The
integration will occur in many steps. An initial CMS integration test
will be over ethernet (software exercise only). A stand alone system
will be tested first at TIF, then at P5, then we will integrate it with
CMS.

The PVSS software and integration sections are not written up yet. It
would be good to see a clear plan for completing this documentation.
Yes!

**Comments from facilitators related to specific presentations**

**Forward Pixel DCS and DSS Systems (presentation by C. Newsom)**

The detector safety system, as presented, is composed of monitoring the
temperature and humidity of the detector in addition to experiment
utilities, radiation levels, and solenoid magnet status. Status signals
are sent to a PLC which generates trip signals to power supply modules.

It is unknown if the utility and global alarms are provided individually
to the Pixel PLC or as a summated signal. We suggest understanding this
issue so that appropriate reactions can be determined. A written
description of the interlock actions is recommended. It should include
signals to and from the DSS, to and from the chiller and brine systems,
whether interlocks reset automatically or have manual reset. As far as I
know communication to us is only one way over the hard lines. The
summation of lines is not foreseen at this time. The exact makeup of
this part of the system is not yet fully designed. Work in this area is
ongoing.

There are no mechanical type interlocks, Klixon or dry contact relays,
in the system. We suggest that the group conduct a Risk Assessment with
reference to the Safety Instrumented Systems standards IEC 61511-1, IEC
61511-2, and IEC 61511-3 to determine if the PLC generated trip signals
are adequate. If not, mechanical devices should be included in the
design. It may be appropriate to provide a separate path which allows a
minimal set of hard-wired devices to ensure FPIX safety. The relay
modules do not require PLC input during a failure exercise, only during
setup. These are effectively "mechanical" once setup. In the case of a
DSS signal, if the relay fails the rack power will be killed using the
hardened rack DSS system.

It was stated that remote controlled coolant flow valves were requested
on Pixel detector pipes. It wasn't clear as to whether or not this
request is actually being implemented. We suggest that the mechanism for
controlling the coolant flow be understood. Remote controlled pneumatic
valves are installed in the system. Additional fine flow balancing
valves will be manually adjusted. They will be in a radiation area when
there is beam.

It appears that there is no mechanism to monitor the coolant flow rate
to individual Pixel cooling loops. We suggest investigating how the flow
rate can be monitored such that appropriate individual cooling loops can
be controlled. (At least one facilitator believes that flow rate
monitoring of individual loops has already been planned). There is a
calibrated reduction aperture with differential pressure gauges on each
loop. The purpose of this system is to remotely measure the flow for
each half cylinder.

The correspondence between power supply distribution and cooling loops
needs to be understood. In the event of a coolant leak, one would want
to power down the section of the detector that corresponds to the failed
cooling loop. The group should confirm the capability of controlling the
appropriate power supplies based on the planned electronics crate
distribution. This has been decided during the last week. The system
will be divided into 8 sections, each independently interlocked.

A clear understanding of the dry gas, warm gas, and sniffer systems
would be beneficial to determine if they are possible monitoring
sources. For example, is it possible to use a sniffer pipe to measure
Fluorocarbons? The dry gas system is now better understood. We have
requested a sniffer and it has been denied. We were invited to pay for a
sniffer if we can find one that will work. No funds are available for
this.

Only a few details of the nitrogen purge system were presented. Failure
modes and back-up systems should be understood and taken into account.
In particular, we think there should be a more complete understanding of
the provisions to ensure that FPIX will remain above dew point during an
unplanned or extended power outage. Would FPIX be warmed? If so, how
would that be accomplished?

The humidity of the incoming gas to the CMS detector is monitored. High
humidity triggers a DSS line to shut down all systems. Failure of the
flow also triggers the DSS. It seems there is no other internal
monitoring, at least at the DSS level. The current plan is to turn off
the power of the FPIX and rely on the heat shield, which is multiply
redundant, to maintain a cold state.

How have the humidity (HMX) and temperature (RTD) sensors that will
actually be used in the detector been tested?

Yes, multiple times.

It is unclear whether the planned number of HMX is sufficient. The
diagram shows their placement in an arbitrary location, far from
sensors. It seems as if the tracker components have many more HMX's than
this. At a minimum, a second HMX could be installed in each volume to
provide some redundancy.

They misunderstood a statement by Christian. I tried to correct it but
apparently was unsuccessful. We have 2 HMX per cylinder.

As mentioned in the talk, the cooling plant circulating pumps have no
backup and will shut down during maintenance or during failures. Since
the FC-72 coolant has a high coefficient of thermal expansion, steps
must be taken to ensure that this fluid can expand as necessary without
damage to the FPIX system if the cooling circulation stops.

This is really important. Simon should ask Paola for the pressure relief
valve settings for our particular lines.

It is mentioned that the FPIX PLC has a battery backup. For how long can
this backup provide power?

Not long. About 15 minutes. We have not personally tested this for our
system.

The NIM article states that the effect of thermal cycling on the
plaquettes has been extensively tested and is under control. How about
effects of humidity changes? Is humidity only an issue if condensation
occurs, or can damage or performance change even without condensation?
Condensation is the only thing that is the biggest danger. HV breakdown
has never been observed in our systems. We can examine the question of
HV beakdown further.

A sketch of the interlock hardware architecture was presented and
reference was made to docdb #1136 which covers PLC hardware. The
interlocks described were to have sixteen sensors wired directly to the
PLC, and 16,000 temperature sensors read through the DAQ and fed to the
PLC, where they would be used for interlocking, perhaps with voting
software. We recommend that the number of sensors used to trip power be
limited. A very large numbers of interlocked sensors will not improve
the reliability but will increase the amount of nuisance trips. This is
something we are very aware of. Our main line of defense are the hard
wired RTDs.

A plan for controlling interlock changes and bypassing is not yet
written. CMS might have such a plan. We will investigate further as
mentioned above.

Information from the TIF interlock failure should be gathered to make
sure the FPIX interlock system does not have the same problem. The
system was not even close to being fully operational when they tried to
use it. It had not even been tested with real hardware when the first
installation/DCS expert had to return to Fermilab. It now is.

**Critical Vulnerability & Forward Pixel DCS Alarms (presentation by L.
Cremaldi)**

Local and Global Alarm and Interlock tables are well underway. We
suggest that the tables be expanded to include subsequent steps required
to recover from cited problems. Perhaps a flow chart based on a matrix
of the tables would be useful. An FMEA for each component in the FPIX
and circulation system is recommended.

*We will begin with a flow chart. I may be ambitious enough to write a
simulator. With adequate manpower support an FMEA analysis could be
contracted.*

Interlocks may be based on absolute temperatures, coolant level, etc.,
or may be based on rate-of-change. It was stated that FPIX would run at
37C with cooling off but with heat shield on. Also that silicon damage
would occur at 20C after radiation. A document defining all trip points
is recommended.

*Yes some of the above absolute temperature limits are being understood
in tests at SiDet and CERN. We need to further document the 37C and 20C
limits. Actually this data is in error. The actual number is 27C, or 37C
above the heat shield temperature of -10C. The point is valid, however.*

Careful development, review, and testing of interlock logic are critical
to ensuring safe FPIX operation. Maximum and minimum design limits for
temperature, pressure, and flow throughout the system should be
determined and taken into account.

*As the PLC code is further developed these trip points will have to be
determined. The DSS has some protections on the cooling plant and N2
flush. We need to determine if these also ensure safe operations for the
pixel detector and implement other fail-safes in our DCS.*

Interlock and alarm functions, operational sequences, and recovery from
an interlock interrupt should be taken into account for all system
modes, including warm start-up, cooldown, cold start-up, cold operation,
warm-up, and servicing.

*Sequenccs for all imaginable detector operations will have to be will
have to be written.*

*You have given a core set above. We fully intend to comply.*

A decision tree should be prepared providing steps that will be taken
if, for example, the barrel pixels DCS system detects a problem even if
the FPIX system does not.

*This is possible and we need to think through the scenarios.*

Conversely, we should plan to communicate the FPIX DCS readings to the
other sub-detectors as well. Finally, we should prepare for the
possibility that cooling conditions of other sub-detectors might impact
some FPIX alarm or interlock set points.

*Yes we have been operating mostly in our Pixel space. Some interaction
has already occurred and is expected to grow. This should be
formalized.*