# Presentation: 921233

## Signal and Image Processing at

- LAPI / POLITEHNICA
**1990 - 2005**
- Vasile Buzuloiu

## “LABORATORUL DE ANALIZA SI PRELUCRAREA IMAGINILOR”

- LAPI stands for
- “**L**ABORATORUL DE **A**NALIZA SI **P**RELUCRAREA **I**MAGINILOR”
- (Image Processing and Analysis Laboratory)
- from the FACULTY OF ELECTRONICS, TELECOMMUNICATIONS and INFORMATION TECHNOLOGY of the
- UNIVERSITATEA POLITEHNICA BUCURESTI (UPB)

## 	The expertise we have at LAPI in the field of (uni-and-

- 	The expertise we have at LAPI in the field of (uni-and-
- multidimensional) **signal processing and statistical analysis** as well as in **software and algorithm development **has allowed us a welcomed participation in TDAQ second level triggering for the future ATLAS experiment of LHC at CERN

## LAPI pre-history (1975 - 1985)

-  involvement in building image analysis systems from scratch (including video ADC/DAC, memories etc)
-  applications of our systems : remote sensing maps analysis, geological microscopic image analysis, dental X-ray image analysis, image compression methods)
-  an award of the Romanian Academy crowned the activity and also stopped it

## LAPI   history (1990 - 2005)

- old age (1990 - 2000)
- Involvement at CERTN in R&D activity for the future detectors of LHC in the framework of the LAA project (led by Prof A. Zichichi) more specifically in the group of Dr. R.K. Bock and W Krischer. Also collaboration with Dr. A. Marchioro for the first level triggering.

## LAPI   history (1990-2005)

- old age (1990- 2000)   (cont.)
- Some results:
-  The ELOISATRON international workshop (in Erice) dedicated to "Image processing for future High Energy Physics Detectors“ (Nov 1991) (V. Buzuloiu director of the workshop and editor of the proceedings)
-  Fast and precise peak finder for the first level triggering (published in the proceedings of the CHEP 1992 - Annecy)
-  4 participants from UPB over various periods between 1993 and 2000

## LAPI   history (1990-2005)

- new age (2000 - 2005)
- The LAPI team starts cooperating with the group of Prof. Bob Dobinson - B. Martin working on:
-  High speed networking (10 Gb/s Ethernet) and long haul intercontinental connections
-  Quality degradation in networks
-  TDAQ for the Second Level Triggering of ATLAS

## LAPI   history (1990-2005)

- new age (2000- 2005)   (cont.)
- Research and teaching home and elsewhere :
-  cooperation with CERN on the same subjects
-  research on various themes of image processing and analysis: filtering, enhancement, image data bases with content indexing, color and multi-component image processing, head tracking, eye tracking, digital photography tools, medical image analysis etc.

## LAPI   history (1990-2005)

- new age (2000- 2005)
- Research and teaching home and elsewhere :   (cont.)
-  R&D in cooperation with foreign universities (Ireland, France, Germany, Italy). Exchange of professors and students
-  R&D in cooperation with foreign companies as e.g. for digital photography tools and eye tracking
-  A yearly international postgraduate, intensive school on multidimensional signal processing and analysis

## The ATLAS TDAQ System

**Notes:**
The particle bunch crossing in the ATLAS detector occurs at approximately 40 MHz. For each bunch crossing the detector records a 1.5 Mbytes “image”, called event.
A hardware Level1 Trigger is used to reduce the data rate from 40 MHz to approx 100 kHz.
The Level2 Trigger is implemented with PCs, interconnected by the DataCollection network. It further reduces the data rate to ~3kHz.
The Level 3 Trigger is again implemented with PC farms (Event processing Farms = EF). It outputs approximately 300 Mbytes/s to mass storage

## ATLAS DataCollection network

**Notes:**
There are 1600 ROBs in the DataCollection network.
3 ROBs are hosted on a PCI card, called RobIn.
Detector Data from the ROBs can be accessed on one of the following  paths:
	-PCI interface of the RobIn
	-Gigabit Ethernet interface of the RobIn
Each individual ROB buffers “Event fragments”, i.e. only part of the data resulting after a collision.

The ROS (Read Out system) represents a group of RobIns:
	-switch based ROS: The robins are hosted in PCs, but the data is read via the GE interface. A number of RobIns is grouped, and connected via a concentrating switch to the central part of the network
	-bus based ROS: The Robins are hosted in PCs, and the data is read via the PCI interface. The PC hosting the RobIns does all the I/O with the central part of the DataCollection network

## DataCollection components

| ROB | 1600 | Read Out Buffer. Stores the event fragments validated by the Level1 Trigger. |
| --- | --- | --- |
| RobIn | 534 | PCI card with 1 GE interface and input from 3 ROLs (Read Out Links). The ROLs feed validated event data to the ROBs. |
| ROS (sb) | 20 | Read Out System. Industrial PCs housing 10 RobIns. The RobIns are configured and monitored via PCI, while the data is read out via GE. |
| ROS (bb) | 136 | Read Out System. High end PCs housing 4 RobIns. The RobIns are configured, monitored and read out via PCI. Data is shipped further by the PC via 2 GE interfaces. |
| L2SV | 10 | Level 2 Supervisor. Supervises a subset of L2PUs, including algorithms for even load balancing between the L2PUs. |
| L2PU | 500 | Level 2 Processing Unit. Runs physics algorithms on a subset of the event data (RoI) indicated by the Level 1 trigger, and validates/invalidates events. |
| PROS | 1 | Pseudo ROS (Read Out System). Temporarily stores the result of the L2 analysis for validated events.  |
| DFM | 35 | Data Flow Manager.  Assigns SFIs for building events using a load balancing algorithm. |
| SFI | 92 | Sub Farm Interface. The events validated by L2 are built together inside the SFI, from all the ROSes. |

**Notes:**
Back-up slide. Contains acronyms explanation and numbers for each components.

## No packet loss!

-  The main data flow is based on a request–response mechanism.
- Message loss implies time-outs and retries at the application level, having a great penalty on its performance.
- TCP guarantees no message loss even if packet loss occurs in the network, but...
- TCP does not scale with large number of connections (~600).
- TCP is CPU intensive.
- TCP increases the packet rate in the network.
- UDP is the preferred protocol for the main dataflow. Packet loss in the network translates into message loss, therefore:
- Special care should be taken in order to minimize packet loss.
- We must avoid multicast/broadcast traffic, as its rate is not guaranteed on all switches.

## Understanding congestion

-  Paper calculation based on queuing theory:
- safe average link utilization as function of buffer size.
- Measurements on switches from several manufacturers with various traffic patterns:
- Throughput / packet loss
- Latency
- Multicast/broadcast handling.
- Measurements performed on a 10-20% size system.
- Computer modeling of the system:
- Model validation by crosschecking with measurements
- Model prediction for full size system

## Contribution to the ATLAS TDAQ system

-  The issues related to the use of Ethernet as network technology have been analyzed and tested on several Ethernet switching devices.
- Gigabit Ethernet traffic generators have been programmed to emulate the output from the level one trigger.
- The large number of these devices allowed us to optimize not only the network topology, but also the traffic patterns and traffic shaping techniques inside the system.
- Thus the performance of the DataFlow has been improved in several steps, and the scalability of the system has been proved up to one tenth of its final size.
- The experience acquired in running the DataFlow system enabled us to derive the requirements for the networking equipment needed in the final system.
- The Gigabit Ethernet traffic generators are currently used in evaluating the performance of sample devices from the major manufacturers.

## The GETB platform

-  Atlas TDAQ DataFlow network
- Large high-speed Layer 2 network (~700 nodes)
- Central switches:  ~250 ports, chassis based
- Concentrating switches: 24 - 48 ports, pizza box
- Packet loss or excessive latency  performance drop
- Specific performance requirements for DataFlow switches
- Need to evaluate devices with realistic test scenarios
- See the “Switch Features” document
- Testing equipment on the market is not fully adequate
- Cost per channel is high
- Not enough flexibility in defining traffic patterns
- Layer 4 - 7 functionality – not essential for Atlas

**Notes:**

Now... let’s see why do we need a new switch tester for Atlas.

First of all because the Atlas Data Acquisition Network is a large flat network 
containing a few hundred nodes which are interconnected using only Layer2 
switching devices. Any problem inside the network will translate into a 
performance drop.

This is why out group defined a set of specific features and requirements that are
to be satisfied by the switches inside the Atlas network. They are all contained in our 
“Switch Features”  document.

While working on this document, we understood that we have to evaluate the 
devices with realistic Atlas traffic scenarios. As this was not possible using commercial 
testing equipment, we decided to create our new platform.
....... As this was not really possible using the commercial equipment available on
the market, we decided to create our own platform.

## GETB Platform – Hardware

- FPGA Firmware
- 90% Handel-C,  10% VHDL
- Commercial IP cores (IP = Intellectual Property)
- Gigabit Ethernet MAC
- PCI Controller

- Logic utilization – approx 85 - 90%
- Single FPGA controls 2 Eth ports
- Multiple projects using the GETB
- GETB Tester, Network Emulator, ROS Emulator
- Common firmware / control infrastructure

**Notes:**
This new platform is called GETB – which stands for Gigabit Ethernet Testbed.
Its basic building block is this PCI card that you can see on the screen. 

The card has 2 Gigabit Ethernet ports and we are using many of these cards to achieve high 
port density.

At the core of the card we have an FPGA – an Altera Stratix device – which drives the 
2 Gigabit ports and controls all the resources available. 
Each port has its own dedicated memory for storing traffic patterns and statistics. 

The FPGA firmware is written in Handel-C, a hardware description language, like VHDL. Inside
the FPGA we are using commercial IP cores for the MAC and PCI interfaces. 

The GETB platform is currently being used in 3 different projects – the GETB Tester which is
the scope of this presentation, a Network Emulator and an Atlas ROS Emulator.

All these projects share a common firmware base and they are all using the same
control infrastructure – as you will see in the next slide. 

## GETB Platform – Control

- Server (runs on Linux)
- Configures the cards
- Monitors status / collects statistics
- Handles remote client requests
- Accesses cards using IO-RCC (DataFlow package)

- Client (runs on any platform)
- Talks to servers using XML-RPC
- Runs user-defined scripts
- Displays statistics in a GUI
- Manages cards from multiple servers

- GETB Client
- (Control PC)

- Device Under Test
- (DUT)

- GETB Servers

- GPS Clock Distribution

- Gigabit Ethernet Links

- Distributed System
- 15 servers hosting 65 cards
- Entirely based on Python

- DUT Control

**Notes:**
The control software of the GETB manages the large scale Testbed. 

It is a distributed system entirely based on the Python language. This has 
many advantages, among them being its scripting facilities and the portability.

The control application has 2 components – the server running on the machines 
hosting the cards and the client, running on the management workstation. 

The server application is responsible for the configuration and constant monitoring 
of the GETB cards. It also allows remote access to the cards.

The client application is the actual interface to the user. It allows the user to control 
any port in any card inside the system. User scripts are executed 
on the client and results are collected and analyzed. 

You will see next a photo of our GETB Testbed. 

## GETB platform - summary

-  The GETB platform provides a flexible environment for the design and development of Gigabit Ethernet applications
- A tester able to evaluate switches for the DataFlow network has been created
- 128 ports running at line-speed
- Client-server traffic emulation provides a way to test devices under realistic Atlas conditions
- Evaluating the device in a worst-case scenario
- We have a comprehensive set of test procedures to check that devices meet our requirements

**Notes:**

So in conclusion we are confident in our new platform, the GETB, that it provides a 
solid base for the development of Gigabit Ethernet applications for Atlas.

We developed a switch tester that is currently being used
to evaluate devices for the DataFlow network.

The Atlas Client Server feature of the tester allows us to stress the equipment 
with realistic traffic scenarios and to obtain results not possible otherwise. 

And we have a comprehensive set of requirements and test procedures that we 
are using to check the Atlas compliance.

## Quality degradation in networks

-  Measure the quality degradation introduced by the building blocks of any network (i.e. switches, routers):
-  emergent properties of traffic differentiation and scheduling mechanisms
-  service fairness for traffic flows of same class/priority
-  Study the relationship between quality degradation in networks and application-level user-perceived quality
-  for Internet Telephony (VoIP) applications
-  for bulk data transfers (FTP or TCP/IP)

## Network Emulation

-  A high-performance network emulator is currently being built by two LAPI researchers, in collaboration with Predictable Network Solutions, Ltd. :
-  capable of reproducing realistic network conditions (correlated loss and delay, natural induced delay variation)
-  implemented on a custom-design FPGA-based platform
-  Emulate the quality degradation likely to appear in large networks
-  emulate the presence of background traffic as vacations of the service facility
-  use a “degradation algebra” to aggregate basic models (of queues and wires) into one single model

## Long-distance networking

- Transmission of native 10 Gigabit Ethernet over the installed base of long-distance telecommunication networks – through the ESTA EU project
- First transatlantic connection based on the 10 GE WAN PHY technology: Oct. 2003, Geneva-Ottawa, live demo at the ITU Telecom World Conference
- Studies for the use of remote computing farms, in real time, in the ATLAS TDAQ system
- Proof of concept in 2004, on a testbed located in Switzerland, Canada, the United Kingdom, Denmark and Poland
- the ATLAS online applications support a remote farm scenario
- the network connections can assure the bandwidth for this particular application

## High-speed signal integrity

- Studies of signal integrity over high-speed backplanes for telecommunication equipment – the ESTA EU project
- Simulating the behavior of different materials and connectors through mathematical models
- Developed the software to control and operate a backplane tester device
- Measurements on real hardware and reconciliation with the simulation models