![](media/image1.wmf){width="5.607638888888889in" height="0.85in"}

CKM PROJECT

CKM DAQ System

CKM Note \#

Rev 0.1

February 24, 2004

CKM Electronics Group

Table of Contents

[1 Overview 3](#overview)

[2 Clock System 5](#clock-system)

[2.1 Functional Description 5](#functional-description)

[2.2 Prototype Efforts 5](#prototype-efforts)

[2.3 Cost 5](#cost)

[2.3.1 Cost of the prototype card 5](#cost-of-the-prototype-card)

[2.3.2 Cost of the entire clock system
5](#cost-of-the-entire-clock-system)

[3 Data Concentration Stage 5](#data-concentration-stage)

[3.1 Functional Description 5](#functional-description-1)

[3.2 Hardware Design Considerations 6](#hardware-design-considerations)

[3.3 Cost 7](#cost-1)

[4 The DAQ Switch Array 8](#the-daq-switch-array)

[4.1 Functional Description 8](#functional-description-2)

[4.2 Cost 8](#cost-2)

[5 The PC Farm 8](#the-pc-farm)

[5.1 Functional Description 8](#functional-description-3)

[5.2 Cost 8](#cost-3)

[6 Control, Monitoring and Data Storage Interface Systems
8](#control-monitoring-and-data-storage-interface-systems)

[6.1 Functional Description 8](#functional-description-4)

[6.2 Cost 8](#cost-4)

# Overview

The CKM detector signals are digitized in the front-end cards. The
digitized data of all detector hits are sent to the DAQ system. Based on
the GEANT simulation, the total volume of the data is about 50 GB per
spill (assuming 1 spill every 3 seconds with 1 second beam). Figure 1
shows the CKM DAQ system.

The detector data flow through Data Concentration Stage, Switch Array
Stage down to the trigger and DAQ PC Farm. The PC Farm performs the data
selection function. Selected data is collected through the Switch Array
and the Mass Data Storage Interface and sent to FCC for primary data
storage. The selected data are monitored locally in the experiment
counting room. They are further selected and sent to WH9E control room
for online analysis and secondary data storage.

The timing standard required by the TDC and ADC functions is generated
by a clock source given to the Clock Generation System. The clock source
can be either accelerator RF or just a crystal oscillator. The clock is
distributed by the Clock Distribution System to the Data Concentration
Stage and is then further distributed to the front-end cards.

Figure 1. The CKM DAQ System.

# Clock System

This section describes the major features of the CKM clock systems.

## Functional Description

The CKM clock system ...

## Prototype Efforts

The CKM clock system ...

## Cost

The CKM clock system ...

### Cost of the prototype card

The CKM clock system ...

### Cost of the entire clock system

The CKM clock system ...

# Data Concentration Stage

This section describes the major features of CKM data concentration
stage.

## Functional Description

The Data Concentration Stage in the CKM DAQ system provides
inter-connections between the front-end electronics cards, Clock system
and the DAQ switch array. It performs two primary functions: (1) to
distribute clock from the clock system to the front-end cards and (2) to
combine data from the front-end cards and to send to the DAQ switch
array.

A Data Concentrator Module (DCM) is a PCI or similar sized printed
circuit board. One possible configuration of the data concentrator
module has one serial link connector for clock system connection, 8
serial link connectors for front-end boards and a Gigabit Ethernet (GE)
connector to the DAQ switch array. Each of the serial links contains 4
differential pairs, two pairs in each direction. They form
bi-directional links between the front-end cards (or clock system) and
the Data Concentrator Module.

The clock system fans out clock signal via serial links. In the DCM, the
clock is recovered. The recovered clock in the DCM drives the return
link to the clock system. The clock system uses the time difference
between the fan-out and return clocks to measure the cable delay and to
make necessary compensation. In the DCM, the recovered clock is further
fanned out to the front-end modules via the serial links.

In the front-end cards, clock is recovered from the serial link. The
detector signal is digitized in the front-end cards and the digitized
data are sent to the DCM via the serial link.

The DCM collects data from front-end cards together. Data are
temporarily stored in the memory chips in the module. The data flow with
balanced rate is sent out via the Gigabit Ethernet connector to the DAQ
switch array.

The DCM will be installed in the PCI slots of regular desktop PC's (if
the format of the DCM is chosen to be PCI). The PCI bus essentially
provides power voltages to the modules. It also provides debugging and
monitoring functions during R&D and commission phases, or
trouble-shooting during the run time.

The configuration, monitoring functions of the DCM for the normal
operation is provided through the serial links to the clock system. This
will assure that the normal operation of the DCM (and therefore the DAQ
system) is not interfered by the software of the PC hosting the DCM.

## Hardware Design Considerations

The functions that DCM will perform can be easily fitted into today's
FPGA. The devices with about 10,000 logic elements (LE's) should be
sufficient. Consider a data width expander or a multiplexer facing 128
bits, which are the primary portions of this FPGA. Each of the 128-bit
blocks typically can be implemented with less than 256 logic elements or
about 2% of the whole device. About 9 of such blocks are needed in this
FPGA, which will take about 25% of the device.

The FPGA with 10000 LE's is considered as a lower middle size one. Some
examples and current (January 2004) prices are listed in Table 1.

Table 1 FPGA Families

  -------------- ------------------ -------------- ----------- -----------------
  Family         Device             \# of LE's     100-499     Note
                                                   cost        

  Altera Stratix EP1S10F484C6       10,570         \$210       www.arrow.com

  Altera Cyclone EP1C12F256C6       12,060         \$168       www.arrow.com

  Xilinx Vertex  Xc2v1000-5FG256C   10240          \$277       www.digikey.com
  II                                                           
  -------------- ------------------ -------------- ----------- -----------------

All the FPGA families listed above support differential I/O pairs at
least at 600Mbps. The serial-to-parallel conversion can be performed
inside the FPGA device directly.

We choose the data rate of about 600 Mbps per pair between the front-end
cards to the DCM, which is similar as in BTeV TDR. With this assumption,
each front-end card has a peak output throughput of 1200 Mbps. This will
relax the requirement for the buffer size in the front-end cards. On the
other hand, relatively inexpensive cables can be used for the
inter-connections between the front-end cards to the DCM.

The temporary memory on the DCM card must have sufficient size and
write/read throughput capacity.

We assume the spill cycle is 3 seconds with 1-second constant beam. Then
the theoretical maximum volume a Gigabit Ethernet link can output in 3
seconds is about 300 MB. A 256 MB buffer should be able to average out
356 MB of data in this configuration. Note that the 300 MB per spill per
GE link or per DCM is the absolute maximum of the data volume. The data
beyond this limit will be discarded anyway; otherwise they will jam the
switch array in the later stage. The real data rate for a GE link base
on the GEANT simulation is about 125 MB per spill. At this rate, the 256
MB buffer has a large safety room.

The memory on the DCM must be able to handle a maximum peak throughput.
We assume at a given time all 8 inputs of the DCM operate at 1600 Mbps
and the Gigabit Ethernet outputs data from the memory buffer at 1000
Mbps. The total peak I/O rate for the memory buffer is 13800 Mbps. With
a 128-bit data port running at 133 MHz, the required peak rate is
achievable. Memory chips of SDRAM at 133 MHz (Micron MT48LC16M16A2BG-7E,
www.arrow.com) cost about \$100 for 256 MB today.

## Cost

We calculated the costs of the data concentration module and the entire
data concentration stage.

The cost for the DCM is show in Table 2.

Table 2 Cost of the data concentration module (DCM)

  -------------- -------------- -------------- ------- ---------------------
                 Cost each      \# Needed      Total   Note

  FPGA           \$200          1              \$200   EP1S10 or similar

  256 Mb RAM     \$12.5         8              \$100   www.arrow.com

  Other parts                                  \$100   

  PCB and                                      \$100   
  assembly                                             

                                Total =        \$500   
  -------------- -------------- -------------- ------- ---------------------

The cost for the Data Concentration Stage is shown in Table 3.

Table 3 Cost of the entire data concentration stage

  -------------- ----------- ----------------- -------------- --------------
                 Cost each   Number Needed     Total Cost     Note

  DCM            \$500       400               \$200K         

  Recycled PC    \$0         100               \$0            

  GE Cables to               400                              
  Switch Array                                                

  Shelf etc.                                                  

                             Total =           \$250K         
  -------------- ----------- ----------------- -------------- --------------

# The DAQ Switch Array

This section describes the major features of CKM DAQ switch array.

## Functional Description

## Cost

# The PC Farm

This section describes the major features of the CKM PC farm.

## Functional Description

## Cost

# Control, Monitoring and Data Storage Interface Systems

This section describes the major features of the CKM control, monitoring
and data storage interface.

## Functional Description

## Cost