Status Report on the Data Acquisition Hardware

The L3DAQ Group:

Mike Clements, Dave Cutts, Sean Mattingly *Brown University*

Andy Haas, Gordon Watts *University of Washington*

Chris Johnson, Ray Zeller *ZRL*

*January 26, 2001*

**Introduction.** The L3DAQ is based on a group of custom modules[^1]
which allow data blocks to flow from the individual detector electronics
digitization crates over high speed data paths to Level 3 trigger
analysis filter nodes. In essence, the blocks flow from a buffer/driver
board (VBD) in a detector group's VME crate, through a readout
concentrator (VRC) which collects data from a number of individual VBDs,
to a bridge associated with a segment of the L3 filter farm (the Segment
Bridge, SB), and on to an interface card (the L3I) in a specific L3
filter node. Figure 1 is a diagram of this flow.

Above is the direct path for detector data; for a description of the
entire system, including the event block routing hardware (the Event Tag
Generator, or ETG), see our recent papers.[^2]

**The SIB.** Each of the new data path modules shown above (VRC, SB,
L3I) share common functions. They all receive and send data through dual
high-speed serial links, using the new INOVA modules. Given the lateness
of these modules we decided to design a single board which could be the
basis of all three system elements, using changes in their FPGA firmware
to configure them appropriately. This board, which in each of the
applications provides a high-speed serial conduit for the data, we
called the Serial Interface Board (SIB).

Actually the SIB is used rather differently in its three sites. As shown
in Figure 2, the VRC uses INOVA links for data flow from (and to) a
separate interface which contains logic for controlling the VBD readout
as well as the wide connectors for the data cables. The VBD also has a
second pair of INOVA links on which data flows to the Segment Bridge
(and may be returned from another SB). In contrast, each of the boards
in the L3 filter node, a L3I, needs only a single pair of INOVA links,
through which data blocks are received and passed if necessary to
another node. Each SIB contains an I960 which acts as a bridge between
the SIB board memory and the host computer's memory. Data targeted for a
given L3 node is received over a SIB's INOVA link and then deposited by
the i960's DMA into the appropriate location in the filter framework in
the node. Data flow in the Segment Bridge is more complicated, but
suffice it to say that each data block flows through two SIBs, each of
which uses only a single pair of INOVA links, but which are linked
through a common PCI bus. Thus use of the SIB in the Segment Bridges, as
well as in the Level 3 filter nodes, requires only one pair of INOVA
links, in contrast to its use in the VRC, which requires two.

Despite their different functionality, as mentioned above we decided to
build a single board, which could be configured through firmware changes
for the different applications. It is relevant to mention that the SIB
is designed as a PCI card, and has an onboard i960 processor which
serves as a bridge between the control logic on the board and the host
PC's bus. Firmware changes and upgrades in the SIB logic can be made in
situ through the PC and even remotely via the network (when authorized).
Additionally, the i960 can serve as a DMA engine to transfer data blocks
into PCI memory: a diagnostic tool for the VRC, where the blocks flow
normally only through the hardware, but the primary data path in the
filter nodes for blocks of events so targeted. We actually plan to have
four SIBs in each L3 filter node, so by far the most boards are used in
this application. Such drove the plan to use a standard size PCI card
for the SIB, so that special housing wouldn't be necessary.

**Troubles.** We are unable to build the SIB board as we planned. A
green light from a board fabricator in December proved illusory, and we
have not been able to converge since then. Basically, the board is
extremely complex; it has proved to be impossible to construct it on the
standard size PCI board.

**Strategy.** Supported by some consultations last week, we dropped the
plan to make a single SIB board for all applications. The fastest thing
that can be done, to get essential hardware to D0, is to use the
existing design, but to remove the output INOVA links. With this change,
this interim board will retain all its VBD readout capability, but the
blocks flowing into the SIB will continue over the i960 bridge into the
host PC, and thence over the network as at present, to the node
emulating the Segment Bridge/ETG. Such in fact was our planned direction
for evolving the DAQ system. It uses the software in place already, in
the "SB/ETG", which in fact can operate unchanged. Since for this time
Ethernet speed will be a limiting factor, we are looking into network
upgrades for the system.

Beyond this interim period we will need, for the VRCs, a non-standard
size PCI card that supports both links as well as, for the SBs and L3Is,
a revised standard PCI card that supports the single pair of links. All
the design elements of course have existed for a long time, so these are
not two new boards. We should also mention that there remain several
special purpose components in the system, the Event Tag Generator and
Event Tag Interface boards, which together enable the Segment Bridges'
event routing capability.

With the arrival of the first boards an extensive software effort will
ramp up, to create the direct control code needed for DAQ operation at
D0. Some work is already done, with the existing SIB driver; but the
real hardware will undoubtedly bring challenges and opportunities. To
exploit fully all the capabilities will take months, but we hope that
basic data flow can be accomplished in a few weeks after receipt of the
hardware.

We have been asked, along with all the other trigger systems, to develop
a completely new schedule, for presentation at the Trigger Meeting on
Thursday 2/1/01. We will make this available on our web site.

[^1]: Please see references linked from our web page at
    *d0l3dev.fnal.gov* reachable also through D0's web. Clearly some
    specifics of the system have evolved, though the papers over 5 years
    are very similar.

[^2]: ^1^