# Presentation: 131355

## SVT Crate Crashes and SPYMON – Status n. 0

_**DAQ meeting 3/13/02**_
**Marco Rescigno**

## Outline

- SPYMON goals and background
- SPYMON structure
- Crate process(monitor)
- Spy manager
- The (G)UI
- The Viewer

- SVT Crate crashes problem
- Things done so far

## SPYMON goal

- SPYMON will monitor SVT performance mostly by looking at Spy-Buffers
- “Spy Buffer can be read out via VME without interfering with the event processing” (not really true, see later)
- Accessing these data can be very useful to catch various potential problems:
- SVT hardware faults
- Cable check
- Board level failures
- Data transmission
- SVXII strips noise
- Inappropriate SVT parameters (e.g. wrong beam position)

## SPYMON goal

- Spy-buffer monitoring program will implement different tasks and may or may not be composed of different subprograms:
- collect statistics of occupancy - SVXII noise monitor (?)
- check non-severe errors and count/display them
- Collect status data from each board and display them
- validate data through hardware emulation
- use track parameters from the last merger to act as a (very fast) beam monitor (?)
- Expert tools for debugging and analysing spy data

## SPYMON vs. TRIGMON

- There is some overlap in scope between Spy Buffer analysis and TRIGMON analysis off L3 data e.g.:
- SVXII hits monitored elsewhere (but noise could affect differently SVT and SVX readout)
- Could check hardware comparing input and output of SVT (but intermediate step crucial for a useful diagnostic )
-  SVT error info are propagated through End Event words to Level2 (but not easy to understand were error originate)
- Moreover SPYMON can access all level1 events
- unbiased statistic (i.e.  level2 reject and level2 accept)
- greater rate (i.e. faster diagnostic)

## SPYMON Structure

- SVT crates
- SPYMON

- Any Unix
- SPYgui

- b0dap68
- SVTSPYMON

- b0dap68
- Spy_manager

- Unix
- HistoDisplayMain

- Disk Files
- Spydumps &
- Root_files

- ACNET
- INTERFACE

- xmlrpc

- TSocket

- SmartSockets

## Early example of system problems : the FISION bug

- We discovered a bug potentially dangerous for every system:
- Two tasks running at the same time on a single crate controller could perform wrong vme I/O due to a bug in the semaphore handling inside FISION
- FISION version 2-15 addressed this problem (Thanks Jim and Yuji).

**Notes:**
Jim complained about slowing performances....

## Further Study on VME I/O

- The most important point for the peaceful coexistence of DAQ process and SVTMON is to let the former access the VME bus (to set DONE register on the TRACER) with a max latency of  0.5 ms (this being the longest time a typical event is readout from the slowest crate).
-  Beyond this limit unwanted dead time would be introduced by SPYMON.
- To handle this we could split spy-buffer readout in many smaller read operation, but how much will it cost?

## VME I/O optimization 1

## VME I/O Optimization 2

## VME I/O Optimization 3

- If RC has to be there we might have to be smart enough
- VME readout speed for crate process depends on L2 rate with an acceptable degradation up to the design 300 Hz

| A | 128 K  | 64 K | 1K | 512 |
| --- | --- | --- | --- | --- |
| 1 KHz | 915 | 945 | 995 | 1KHz |
| 300 Hz | 281 |  |  | 300 |
| 75 Hz | 74 |  |  | 75 |

| B | 128 K  | 64 K | 1K | 512 |
| --- | --- | --- | --- | --- |
| 1 KHz | 85 ms | 124 ms | 159 ms | 160 ms |
| 300 Hz | 74 ms |  |  |  81ms |
| 0 Hz | 69 ms |  |  | 71 ms |

**Effective L2 rate and (B) VME average I/O speed **
**(MRG-OUT) vs. nominal L2 rate and packet size**

## SVT crate crashes

- Unfortunately a long standing problem (it is not new)
- When crate hangs is impossible to ping/vxlogin/minicom to it... (this make hard to debug)
- Crashes observed in all the 8 crates
- Rate of  crashes ~1 week per crate
- SPYMON is never started/stopped for all the period the crate is alive!
- Crashes seem to occur only during data taking

## Checks already performed

- SPYMON does not leaks (total memory usage is constant)
- One type of crashes observed in the past was related to CPU memory fragmentation (poor pool memory management in VxWorks) this is not any more an issue with newer CPU’s with more memory (bought by Rome group, installed during shutdown)
- Expanded printout from SPYMON to understand possible correlation with tail in timing performance of the program and to have hints on where the problem arise

## Checks already performed

- Some hints that the crashes occur when large smart-messages are sent by SPYMON
- Reduced max message sizes
- typically now the message is split in 10 blocks of ~16 Kb
- Spydump message still unsplit (need wider modification, under way)  presently is 150 Kb
- The above is also needed to allow user selection of buffer to dump

## What will be attempter next

- Arnd had suggested to sent other messages from the svt crate though SPYMON RT connection
- Some test done yesterday (only a partial success I crashed Readout but messages sent during coldstart are correctly delivered)
- We are ready to limit the spydumps only when SVT detects some error (this can be implemented right away)
- This should help keep the crate crashes well below the 1 per week rate

## Conclusion?....

- This is currently the highest priority problem I have to solve
- With some help (Arnd?) we can have all the messages going through only one connection soon
-  but...
- SPYMON has been and will be a fundamental tool for SVT debugging/operation ... don’t shut it down!