GENERAL TECHNICAL ASSESSMENT OF THE PROPOSED BTeV

PIXEL-BASED TRIGGER SYSTEM.

\* \* \* \* \* \* \* \*

June 12, 2000

### R. Tschirhart, P. Wilson Fermilab.

We have been charged by the Directorate to make a general assessment
regarding the technical feasibility of the pixel-based detached vertex
trigger scheme presented in the BTeV proposal. This assessment is
intended as support material for Fermilab PAC deliberations of the BTeV
proposal. Materials for this assessment included the BTeV proposal text,
as well as a total of about seven hours of meetings with BTeV proponents
where approximately twenty technical discussion points were addressed in
detail. In addition we have had some discussion with Beams Division
staff regarding the projected beam conditions at the C0 interaction
region. We appreciate the patience of BTeV proponents and the
forthcoming nature of our discussions. Additional information relevant
to this assessment was collected during the cost review of the BTeV WBS.

The BTeV pixel-based detached vertex trigger concept is an ambitious and
powerful scheme that proposes to simultaneously exploit the pattern
recognition potential of pixel detectors and the ever-increasing muscle
of commercially available processors, memory, firmware and software. It
is clear that success requires high performance on all these fronts,
where "success" means the following:

1\) Reject 99.97% of the non-B interactions in real-time at a mean
luminosity of 2\*10\*\*32/cm\*\*2-sec

2\) Maintain a trigger acceptance of 50% or more over a broad range of
B-decays at a mean luminosity of 2\*10\*\*32/cm\*\*2-sec. (Trigger
acceptance here is normalized to the offline B-reconstruction
efficiency.)

3\) An online trigger system that is highly "self-aware", that is, where
the integrity and content of the data is continuously monitored.

BTeV proposes to achieve the above performance with a 3-level trigger
system where the Level-1 trigger operates at the 108-bunch crossing rate
(132ns spacing) and provides the bulk (\~x100) of the rejection. The
technical risk of the Level-1 trigger will be assessed below. The
Level-2 and Level-3 triggers are implemented in commercial processor
farms where essentially the offline reconstruction code is running as a
filter on the data. Level-2 is distinguished from Level-3 by how much of
the event data is used to process the event. Level-2 is proposed to
process the event based on a refined analysis of the pixel data and some
additional tracking data that is dynamically acquired event by event.
Level-3 uses data from all detector elements to process the event. At
this point it is not clear as to how much of the event data is used by
Level-2 for a trigger decision. If more than about 1/3 of the event data
must be available to Level-2 on average, then it might make more sense
to remove Level-2 (x,y)from the architecture and enlarge the size of the
Level-3 system where the full event is processed. Such a move has the
significant advantage of reducing the complexity of the overall system.
The BTeV proponents are weighing these Level-2/Level-3 architectural
issues, and we urge them to assign a substantial weight to reducing the
complexity of the system at the cost of increasing the size of the
Level-3 farm.

Although the Level-2/Level-3 filters present a significant challenge,
the recent experience with large scale farms suggest that a
Level-2/Level-3 system the size that BTeV proposes can probably be
achieved with about the resources described in the proposal and WBS
documents. BTeV has assigned a relatively large cost contingency (100%)
for processors that make up the Level-3 system, which results from the
current uncertainty of how much processing is required at Level-3 to
reduce the final stored data volume to a manageable level. The
understanding of the Level-3 processing needs will no doubt improve as
the trigger/filter algorithms are better understood and the date of
processor purchase gets closer.

The success of the Level-2/Level-3 and ultimately that of the experiment
hinges on the performance of the pixel-based Level-1 trigger. It's clear
that in principle a pixel detector of the type that BTeV is proposing
can dramatically ease the pattern recognition burden online and offline.
The experience with pixel prototypes in the BTeV test-beam has been
quite positive, and the studies of the pixel resolution function
presented in the proposal are particularly impressive. As to whether
these promising results can eventually evolve into the high Level-1
trigger performance required was the primary focus of our discussions
with BTeV proponents. A summary of the key issues, which are not
restricted to Level-1 concerns, follows. As a mechanism to order our
assessment of technical risk, we will summarize the issues below with a
risk index of 1-10, where 10 is normalized to what we regard as the
greatest technical risk (grading on a curve).

## 1) Physics inputs and accelerator performance inputs.

The simulation of collision activity in the pixel detector and
surrounding detector system is based on a PYTHIA generator followed by
full GEANT3(FLUKA) treatment. In response to our concerns as to how well
the particle fluence is understood in the forward direction, the BTeV
proponents produced published CDF and UA5 data that is reasonably well
described by PYTHIA in the relevant forward regions.

\* \*Technical Risk: 3.

PYTHIA simulations predict that about 1.6% of the forward cross-section
will contain charm. At first blush this might appear to be a mechanism
to swamp a tracking trigger that requires x100 rejection at the outset.
In fact, given that the central B cross-section exceeds expectations by
about a factor of x2, this "charm pollution" could be considerably
higher. The efficiency for charm is quite a bit less however, and if
necessary an effective mass cut can be imposed to ease the burden on the
higher levels of the trigger. There appears to be adequate bandwidth
headroom at Level-1 to deal with even large variations in the charm
fraction.

\* \*Technical Risk: 1.

Regarding the required accelerator performance, we considered the impact
of bunch-to-bunch intensity variations, effect of the attendant bunch RF
pulse, and the stability and size of of the interaction volume. Of these
concerns the effect of the RF pulse on the analog performance of the
pixel device and the beam stability were most significant.

Regarding the effect of the bunch RF pulse, the BTeV proponents have
done some nice work with test structures described in the proposal that
demonstrates that the digital circuitry of the pixel readout will likely
be able to tolerate the RF pulse. Similar work remains to be done on the
very sensitive analog components of the readout. BTeV proponents
presented plausibility arguments as to why this will not be a problem,
as well as referring to the successful noise performance of the HERA-B
silicon system. These arguments and experiences notwithstanding, we
regard this point as a risk concern.

\* Technical Risk: 2.

Regarding the stability and size of the interaction volume, it became
clear during our discussions that the current online vertexing algorithm
requires that the transverse (x,y) size of the beam volume must remain
well within a 250 micron radius. This requirement stems from the first
stage of the primary vertex reconstruction that only considers tracks
that pass within 250 microns of the beam transverse centroid, assumed at
(0,0). BTeV chose this parameter based on assurances from the Beams
Division that achieving a spot size of 50 microns (RMS) with +/-100
micron control of the centroid would be straightforward. This algorithm
may in fact evolve in the future to be less sensitive to the beam size
and centroid. This requirement on the beam centroid also implies a
requirement on the angle of beam relative to Pixels at level of
250-micron/1-m or 250 micro-radians.

The BTeV beam position requirements are less stringent than those for
the CDF Silicon Vertex Trigger. In the CDF Trigger, the primary vertex
is not reconstructed and the trigger cuts on the impact parameter
relative to an assumed beam position. Based on CDF measurements from
Run-I, the beam position is stable within a store to about 50 microns in
the transverse directions and 150 micro-radians in angle. Over a period
of months the variations observed were much larger: several millimeters
in the transverse direction and 500 micro radians. To achieve the
required stability of the beam in Run II, CDF will be installing an
active feedback system to the Beams IR control. Although the
requirements for BTeV are less stringent, the prime trigger for BTeV is
the impact parameter at L1 so the need to achieve this stability is a
high priority.

\*Technical Risk: 2, ASSUMING the Beams Division will be able to control
the the beam centroid and volume to the required level.

## 2) Level-1 trigger architecture and algorithms.

The Level 1 trigger algorithm is subdivided into a 5 step sequence:

finding clusters of Pixel hits, matching pixel clusters in sets of 3
stations to track segment patterns, linking segments into tracks in a
track fitter, finding primary vertices and finally identifying high
impact parameter tracks for triggering. The first two stages of the
algorithm are to be executed in a parallel fashion with custom logic
using Field Programmable Gate Arrays (FPGAs). The latter stages are to
be executed with custom electronics using a large number of Digital
Signal Processors (DSPs). Since the raw data entering the first stages
of L1 are not time ordered, a fast switch is required to sort the data
between the segment finding and track fitting stages.

### Clustering:

The clustering algorithm is simple and should be easily implemented in
an FPGA as planned. Contiguous readout pixels are clustered together and
a centroid is assigned.

In discussions with the simulation group and the hardware engineers, it
became apparent that the algorithm simulated to date is not exactly the
one that can be practically implemented in hardware. The simulation does
not take account of any physical boundaries between columns in readout
chips or between readout chips. The effect of this will be to find extra
clusters in the real hardware. The number of extra clusters will be
different in the x and y planes because of the effect of the dipole
field but is not expected to be more than about 10% in either case. The
simulation should be updated to include the real boundaries to ensure
there are no unforeseen problems.

The clustering discussion led to a discussion on whether to cluster
across column boundaries within the FPIX2 chip. This would result in a
significant increase in the complexity of the system since the data into
a given cluster would then no longer be time ordered.

\* \*Technical Risk (Clustering Algorithm): 2, assuming no attempt is
made to cluster across column boundaries,

### Segment Finding:

The bulk of the recent development work on the trigger algorithms has
been on the segment finding and track fitting. The segment finder
provides a good reduction in data rate for input to the track
linking/fitting stage while retaining high efficiency for real tracks.
The number of found triplet segments is about a factor of x5 lower than
the number of input pixel clusters. The algorithm is explicitly designed
to be insensitive to the beam position.

Large FPGAs that are available today are well suited to the segment
finding algorithm. The large I/O can be handled with high pin density
packages. Simple calculations doing linear projections can be carried
out quickly. There are also sufficient resources for buffering data on
chip to guard against rate fluctuations.

Timing of the segment finding operation is driven by the number of pixel
clusters that must be processed per quadrant. An estimate of this timing
has been made assuming an average of 2 interactions per crossing and
cluster occupancies from the GEANT simulation. This estimate gives an
average of one event processed every 70ns which is within the absolute
requirement of 132ns per event. Although the margin appears generous,
the consequence of exceeding 132ns are very dear. If the average time
goes to 140ns/event, due to higher than expected occupancies for
example, the front-end will very quickly back up the Level 1 input
buffers. The trigger is presented as having sufficient buffering to
handle the crossing rate as input with no throttle on the input stream
for the possibility of saturation. Such a throttle is important
insurance that would allow the experiment to operate (at reduced
efficiency) while working on fixes for a rate problem.

\* Technical Risk (Segment Finding algorithm): 5

### Track Fitting and Vertexing:

The Track Fitting and Vertexing are to be performed in two farms of
DSPs. Algorithms sufficient for these tasks have been developed and
extensively simulated. The primary hurdle to demonstrating the
feasibility of this scheme is to determine the processing requirements
of the algorithms. The algorithms were coded in C and compiled against a
currently available Texas Instruments DSP. The number of DSP clock
cycles required (which scales the number of DSPs required) for the
compiled C code was considered to be much too large. As a solution, the
assembly language output of the C compiler was optimized by hand. The
optimized code was estimated to require about a factor of x50 fewer
instructions/event than the non-optimized code. The size of the proposed
DSP processor plant includes this factor of x50 presumed speed
improvement.

There are several concerns about this code optimization. During
discussions it became apparent that after optimization the assembly
language code has not been run on a DSP or DSP simulator to demonstrate
that it works. Further, the person who carried out this specialized
optimization is no longer with the BTeV project. We note that this
custom assembly language code would have to be maintained throughout the
life of the experiment. In fact, it is likely that significant revision
of the code would be required in the face of real data from the
experiment.

A final concern is the small margins built in to estimates for the DSP
requirements. A relatively small margin of about 5% is included for
overhead for "operating system". This seems like a very optimistic
overhead assumption. There is no margin factor (above some generous
assumptions on individual sections of code) included in determining the
number of processors needed to carryout the calculations in 132ns. The
need for an input throttle described above applies to the DSP processing
as well.

\* Technical Risk (Track Fitting and Vertex Finding Algorithms): 9

## 3) Simulation studies of the Level-1 trigger.

The design of algorithms for the Level-1 trigger has been driven by
extensive physics simulations using BTeV GEANT package. These physics
simulations are a proof-of-principal for the detached vertex trigger
algorithms. The simulations are well developed and where possible have
been bench marked against real data (e.g. pixel hit rates from test
beam).

The BTeV proposal provides limited information on underlying performance
measures from the simulation. The BTeV proponents provided plots
demonstrating: track momentum resolution, track impact parameter
resolution, and primary vertex resolution (in x, y and z coordinates).
Further, they provided tables indicating event composition of the Level
1 and Level 2 trigger outputs. These measures are consistent with the
100 to 1 rejection of the Level 1 trigger. After the Level 1 trigger the
event composition is approximately 4% b-quark, 10% c-quark, 40% contain
strange particles and the remaining 45% consist of tracks with "false
impact parameter" resulting from various types of detector or
reconstruction effects. An analysis of how daughter tracks from strange
particles are rejected (essentially 100% of crossings contain strange
daughters) was also provided. Two requirements provide much of the
rejection for strange daughters at Level 1: Pt\>0.5 GeV/c rejects \~5/6
tracks and \~6/7 are rejected by the upper limit on impact parameter
(2mm). The majority of the events passing due to "false impact
parameter" are the result of conversions, re-interactions, multiple
scattering and delta rays or more globally interaction with detector
material. This emphasizes the need to keep a close handle on the
material budget of the pixel detector.

Concerns with possible confusion on primary vertex finding at high
luminosity were addressed. The vertex position resolutions were
consistent with vertex separation distributions shown in the proposal.
The study of number of vertices found as a function of average number of
interactions per crossing (ranging from 1 to 6) shows an average of 0.6
vertices found/interaction. The efficiency is this low because these
interactions include elastic and diffractive events which are much less
likely to generate a reconstructible primary. No increase (or decrease)
is seen in the slope indicating no significant problem with extra
vertices found.

\* \*Technical Risk (Physics Simulations): 6

Another type of simulation which is no less critical to the success of
the trigger has seen less attention. These are queuing simulations and
timing studies to ensure that the processing and data pipelines are
capable of handling the onslaught of data. Studies have been performed
on a previous version of the clustering and segment finding algorithm.
The results were described as showing the Level-1 timing was:
"\...workable but not comfortable". Given the extensive changes in the
the Level-1 and Level-2 algorithms since this queuing study was done, it
is critical that timing studies be repeated and expanded.

\* \*Technical Risk (Queuing simulations): 7

## 4) System Robustness and Integrity.

BTeV proponents have studied how the rejection of min-bias events and
acceptance of B-decay events is affected by the addition of random noise
hits (up to x40 of expected) that are distributed flat and peaked at the
beam. The rejection and acceptance show a remarkable robustness against
these noise hits, which no doubt traces back to the pattern recognition
power of a pixel system. In addition, BTeV proponents have studied how
the rejection and acceptance varies as a function of mean luminosity,
and have found that the Acceptance\*Rejection product degrades by about
(40+/-10)% (\~33% rejection loss, \~7% acceptance loss) at twice the
nominal mean of two interactions/crossing. Although this loss is
somewhat uncomfortable, these studies indicate that the trigger
algorithm is not sitting near a cliff of instability.

\* \*Technical Risk (Algorithm stability to noise/luminosity
variations): 3.

Regarding the robustness and integrity of the hardware and software
design of the trigger system, these issues and concerns have only begun
to be addressed at a conceptual level by BTeV proponents. It is clear
that BTeV proponents have focused their resources on developing pixel
detector prototypes and generating a plausibility argument that a
workable trigger algorithm is possible. Given the very complex nature of
this system where thousands of events are simultaneously and
asynchronously cooking, issues of data integrity, robustness, and
monitoring are critically important and have the capacity to cripple a
design if not dealt with at the outset. It is simply a fact of life that
processors and processes die and get corrupted, sometimes in subtle
ways. BTeV has allocated some resources for control and monitoring, but
our assessment is that the current allocation of resources will be
insufficient to supply the necessary level of "self-awareness" in the
trigger system. In particular, the level of physicist involvement in
trigger design and online monitoring is far too low, and needs to be
increased to realize a physics ready system.

\*\*Technical Risk (Error Detection/Correction, Monitoring): 10.

## Summary of Assessment and Manpower Considerations.

It is clear that BTeV proponents have done some excellent work towards
demonstrating that a pixel-based vertex trigger is plausible at Tevatron
luminosities. The pattern recognition power of a pixel-based detector is
quite impressive, which together with BTeV's reasonable estimates of
particle fluence and rates suggest that fast and robust trigger
algorithms should be achievable. However, the previously listed caveats
must be addressed before this is convincingly demonstrated. For the
project to succeed it is equally clear that there is an enormous amount
of work remaining to realize an engineered system. One could argue that
expecting fully engineered solutions is not realistic at the proposal
stage, but it is important not to lose sight of the large amount of
development work summarized above that will be necessary for eventual
success.

The most critical element to eventual success is not addressed above,
and that is the issue of physicist and engineering manpower. This effort
MUST GROW on both fronts, and likely beyond the bounds of required
manpower outlined in the WBS documentation. In particular, the current
level of physicist involvement in design and close interaction with the
engineering effort through all phases of the project must grow
substantially. This is not to make light of the very impressive
physicist and engineering trigger efforts to date, which most recently
have been dominated by Fermilab staff. A system of this complexity and
innovation however would clearly benefit from other strong physicist &
engineering group(s) from outside Fermilab playing central roles as
well. Such group(s) would serve as a critical internal counterbalance
during the design, construction, commissioning and operational phase.
Without an increased pool of design skills and experience to draw from
and thermalize with, the project will remain at risk. The exciting
challenge of designing and building a real life pixel-based trigger
system certainly has the potential to attract additional strong groups.

Please don't hesitate to contact us if you have any questions or
comments regarding this assessment.

Best Regards,

R. Tschirhart, P. Wilson.