# Presentation: 281033

## EVB Simulation Status

**EVBSIM code changes**
**FNAL test stand comparison**
**LHC case studies**
**Ptolemy progress**
**Plans**

## XDAQ simulation

## EVBSIM code changes

**LHC studies use barrel shifting**
**RU for event traffic ******** BU**
**EVM for control traffic ******** BU**
**BU for control traffic ******** EVM, RU**
- Not really needed for 64x64
**Scales nicely at least up to 256x256 (see plot)**
**FNAL test stand studies do NOT have barrel shifting**
**EVBSIM memory consumption**
**Some leaks were found with Insure++**
- It also reported “problems” erroneously
**512x512 memory usage still >~ 2 GB**
- Always crashes on cmsun1

## FNAL NxM setup (1/2)

**XDAQ test stand and EVBSIM measure:**
**Event data bandwidth per BU, in MB/s**
**All BUs  are checked to give the same value**
**Tables give EVBSIM/XDAQ bandwidth ratios**
**For each setup vary 2 parameters:**
**N = number of event IDs per control packet**
- N = 1, 4, 16, 64
**R = number of active event ID request packets per BU**
- R = 1, 4, 16, 64
**N x R = number of active event IDs per BU**
- N x R  256 = number of resources per BU

## FNAL NxM setup (2/2)

**Event fragment 2 kB per RU**
**Control packets: **
**64 bytes/eventID ****(too large?)**** + 224 bytes I2O header**
**Data packets:**
**2 kB event data + 224 bytes I2O header**
**L2 needs all RUs**
**Each event rejected after 0 ms **
**PCI bandwidth 132 MB/s**
**Myrinet bandwidth 160 MB/s**

## FNAL 1xM comparisons (1/2)

**1x1:**
** ****R \ N |        1        |        4        |       16        |       64**
**-------+-----------------+-----------------+-----------------+-----------------**
** ****1     |  8 / 10.7 = 0.7 | 23 / 27.2 = 0.8 | 43 / 43.8 = 1.0 | 56 / 52.0 = 1.1**
** ****4     | 23 / 23.8 = 1.0 | 48 / 49.3 = 1.0 | 62 / 68.0 = 0.9 | 66 / 68.1 = 1.0**
**16     | 25 / 25.2 = 1.0 | 48 / 49.3 = 1.0 | 61 / 64.8 = 0.9**
**64     | **_**25 / 25.4 = 1.0**_
**1x2:**
** ****R \ N |        1        |        4        |       16        |       64**
**-------+-----------------+-----------------+-----------------+-----------------**
** ****1     |  8 / 10.6 = 0.8 | 22 / 26.4 = 0.8 | 35 / 33.7 = 1.0 | 35 / 35.0 = 1.0**
** ****4     | 22 / 22.3 = 1.0 | 32 / 33.9 = 0.9 | 35 / 35.2 = 1.0 | 35 / 34.8 = 1.0**
**16     | 25 / 24.9 = 1.0 | 32 / 34.2 = 0.9 | 35 / 35.2 = 1.0**
**64     | 25 / 25.1 = 1.0**
**1x4:**
** ****R \ N |        1        |        4        |       16        |       64**
**-------+-----------------+-----------------+-----------------+-----------------**
** ****1     |  8 /  9.3 = 0.9 | 16 / 16.6 = 1.0 | 17 / 17.4 = 1.0 | 18 / 17.1 = 1.1**
** ****4     | 12 / 12.7 = 0.9 | 16 / 17.0 = 0.9 | 17 / 17.5 = 1.0 | 18 / 17.1 = 1.1**
**16     | 12 / 12.6 = 1.0 | 16 / 17.2 = 0.9 | 17 / 17.5 = 1.0**
**64     | 12 / 12.6 = 1.0**

## FNAL 1xM comparisons (2/2)

## FNAL 2xM comparisons (1/2)

**2x1:**
** ****R \ N |        1        |        4        |       16        |       64**
**-------+-----------------+-----------------+-----------------+-----------------**
** ****1     | 14 / 18.8 = 0.7 | 34 / 41.5 = 0.8 | 54 / 60.0 = 0.9 | 63 / 67.1 = 0.9**
** ****4     | 34 / 32.6 = 1.0 | 54 / 56.5 = 1.0 | 65 / 71.6 = 0.9 | 68 / 78.0 = 0.9**
**16     | 34 / 33.4 = 1.0 | 55 / 56.5 = 1.0 | 65 / 71.9 = 0.9**
**64     | 34 / 33.6 = 1.0**
**2x2:**
** ****R \ N |        1        |        4        |       16        |       64**
**-------+-----------------+-----------------+-----------------+-----------------**
** ****1     | 14 / 18.7 = 0.7 | 34 / 40.5 = 0.8 | 54 / 53.3 = 1.0 | 63 / 55.0 = 1.1**
** ****4     | 34 / 31.9 = 1.1 | 54 / 55.6 = 1.0 | 65 / 66.0 = 1.0 | 68 / 57.5 = 1.2**
**16     | 34 / 33.0 = 1.0 | 55 / 55.7 = 1.0 | 65 / 66.3 = 1.0**
**64     | 34 / 33.1 = 1.0**
**2x4:**
** ****R \ N |        1        |        4        |       16        |       64**
**-------+-----------------+-----------------+-----------------+-----------------**
** ****1     | 14 / 17.4 = 0.8 | 32 / 32.2 = 1.0 | 35 / 34.9 = 1.0 | 34 / 34.2 = 1.0**
** ****4     | 25 / 26.1 = 1.0 | 32 / 33.8 = 0.9 | 34 / 35.3 = 1.0 | 35 / 29.8 = 1.2**
**16     | 25 / 26.0 = 1.0 | 32 / 33.9 = 0.9 | 34 / 35.2 = 1.0**
**64     | 25 / 25.9 = 1.0**

## FNAL 2xM comparisons (2/2)

## FNAL 4xM comparisons (1/2)

**4x1:**
** ****R \ N |        1        |        4        |       16        |       64**
**-------+-----------------+-----------------+-----------------+-----------------**
** ****1     | 24 / 28.9 = 0.8 | 47 / 53.1 = 0.9 | 62 / 67.6 = 0.9 | 68 / 71.9 = 0.9**
** ****4     | 40 / 39.7 = 1.0 | 59 / 61.3 = 1.0 | 67 / 73.2 = 0.9 | 70 / 78.4 = 0.9**
**16     | 40 / 40.0 = 1.0 | 59 / 61.3 = 1.0 | 67 / 73.1 = 0.9**
**64     | 40 / 40.0 = 1.0**
**4x2:**
** ****R \ N |        1        |        4        |       16        |       64**
**-------+-----------------+-----------------+-----------------+-----------------**
** ****1     | 23 / 28.7 = 0.8 | 47 / 45.1 = 1.0 | 62 / 57.2 = 1.1 | 68 / **_**49.9**_** = 1.4**
** ****4     | 40 / 39.2 = 1.0 | 59 / **_**52.1**_** = 1.1 | 67 / **_**56.0**_** = 1.2 | 69 / 69.7 = 1.0**
**16     | 40 / 39.3 = 1.0 | 59 / **_**52.1**_** = 1.1 | 67 / **_**50.2**_** = 1.3**
**64     | 40 / 39.6 = 1.0**
**4x4:**
** ****R \ N |        1        |        4        |       16        |       64**
**-------+-----------------+-----------------+-----------------+-----------------**
** ****1     | 23 / 27.9 = 0.8 | 45 / 42.4 = 1.1 | 61 / 57.1 = 1.1 | 67 / 57.2 = 1.2**
** ****4     | 40 / 37.5 = 1.1 | 59 / 56.1 = 1.1 | 67 / 62.5 = 1.1 | 69 / 54.8 = 1.3**
**16     | 40 / 39.1 = 1.0 | 59 / 56.4 = 1.0 | 66 / 62.4 = 1.1**
**64     | 40 / 39.1 = 1.0**

## FNAL 4xM comparisons (2/2)

## Latency overheads tuned only for            1x1, R=64, N=1:
Send      DMA latency = 11 + bytes/132 us
Receive DMA latency = 11 + bytes/132 us
Myrinet transfer latency: 1 + bytes/160 us
Overall EVBSIM/XDAQ agreement not bad
Some XDAQ values are strange
They do not agree with the trends
Various values have been shown to be reproducible only within 10% sometimes
Slopes for R=1 not understood

**Send      DMA latency = **_**11**_** + bytes/132 us**
**Receive DMA latency = **_**11**_** + bytes/132 us**
**Myrinet transfer latency: **_**1**_** + bytes/160 us**
**Overall EVBSIM/XDAQ agreement not bad**
**Some XDAQ values are strange**
**They do not agree with the trends**
**Various values have been shown to be reproducible only within 10% **_**sometimes**_
**Slopes for R=1 not understood**

**FNAL *****N*****x****M results**

## Each BU emulates M Filter Nodes (see plots for values)
Each Node has 1 CPU for HLT calculations
Events arrive in zero time from the BU:
A second CPU is assumed to do that in the background
Event fragment 2 kB per RU
Control packets: N=32, R=8
 64 bytes/eventID (too large) + 224 bytes I2O header
Data packets:
 2 kB event data + 224 bytes I2O header
L2 needs all RUs
Each event rejected after X ms (see plots for values)
DMA send latency: 1 + bytes/528 us
Myrinet transfer latency: 0 + bytes/250 us
DMA receive latency: 1 + bytes/528 us

**Each Node has 1 CPU for HLT calculations**
**Events arrive in zero time from the BU:**
- A second CPU is assumed to do that in the background
**Event fragment 2 kB per RU**
**Control packets: N=32, R=8**
** ****64 bytes/eventID ****(too large)**** + 224 bytes I2O header**
**Data packets:**
** ****2 kB event data + 224 bytes I2O header**
**L2 needs all RUs**
**Each event rejected after X ms (****see plots for values****)**
**DMA send latency: 1 + bytes/528 us**
**Myrinet transfer latency: 0 + bytes/250 us**
**DMA receive latency: 1 + bytes/528 us**

**NxN “LHC” tests**

## 64x64 “LHC” test (1/4)

## 64x64 “LHC” test (2/4)

## 64x64 “LHC” test (3/4)

## 64x64 “LHC” test (4/4)

## 256x256 “LHC” test

## Ptolemy status at FNAL

**Started from code by Samim, Alexander and Frans**
**Implemented Builder Manager**
**Extra XDAQ messages**
- BMControl (Allocate, Clear) and Confirm
**BM replaces the last RU**
- Participates nicely in “RU” barrel shifting
**Implemented PCI + barrel shifting in BU**
**Implemented Readout Manager with rudimentary GTP + RCN**

## Ptolemy work at FNAL

**EVM/RU:**
**Do we need detailed GTP/RCN handling?**
**RU:**
**Do we need detailed FED/DDU handling?**
**BU must copy events to Filter Nodes**
**This will double the PCI bus load**
- Check the effect
**Add more ntuples**
**Study plots**
**What else?**

## Plans

**Keep CNCL code mostly frozen**
**We may want to run it occasionally**
**http://home.fnal.gov/~litmaath/evbsimrs.ps:**
- How does EVBSIM work?
- What are its results?
**Update CVS repositories at CERN+FNAL**
**Include example configurations, scripts and resulting numbers and plots**
**Replicate full XDAQ protocol in Ptolemy**
**Compare with CERN+FNAL test stands**
**Repeat scaling studies**