**Review of DZero CTT Upgrade Proposal (4/8/04)**

Darin Acosta, John Anderson, Bill Freeman

# Introduction

# 

The committee is convinced that the upgrade to the L1 Central Track
Trigger (L1CTT) is needed for operation in Run 2b. Without an increase
in the track equations to utilize the full resolution of the fiber
tracker, the fake rate at Run 2b luminosities makes the track trigger
non-functional, compromising physics.

The approved baseline for the L1CTT is an upgraded Digital Front-End
daughter board (DFEA), with larger FPGAs to handle a 3−5X increase in
the track equations. However, the current electronic system is stated to
be untenable: the upgraded DFEA daughter boards in current L1CTT
crate+controller system would require 12 hour downloads to program,
which is deemed very unlikely to succeed during operations. Although the
reviewers question the necessity of frequent downloads in the collision
hall, and whether the downloads could not be made more robust, we will
assume the necessity of a higher bandwidth connection since that is the
central tenet of this upgrade proposal. Other benefits from this upgrade
include improved testability of the system and an improved distribution
of the clock and control signals. Thus, the full scope of the L1CTT
upgrade includes the following:

1)  DFEA daughterboard (approved in baseline)

2)  DFEM motherboard

3)  DFEA backplane

4)  DFEA crate controller

5)  Optical DCL (crate controller interface)

6)  LVDS splitter

As part of the DFEM motherboard project, a DSAT tester module is also
included to allow DFEM/DFEA bench testing without the crate backplane
and controller. In total, the L1CTT system is housed in 2 crates of 20
motherboards each. Each motherboard would contain two DFEA daughter
boards (which may be integrated on-board). Two new crate controllers are
required. Each of the first 5 items is critical to the success of the
L1CTT.

The scheduled completion date for the L1CTT upgrade is June 2005
according to the Project file. The final deadline for completion of the
Run2b project, as mandated by DOE, is April 2006. However, the
proponents of the design would like to take advantage of a shutdown
beginning this summer, and have a subset of the upgraded L1CTT boards
placed in-situ in the hall by September 2004. The upgraded L1CTT would
receive signals in tandem with the current L1CTT using an LVDS splitter
box, allowing the system to be tested with real data in the collision
hall environment.

# Technical Feasibility

The DFEA daughter board development is well advanced, with a prototype
in hand based on the XC2V4000 FPGA and with track equation firmware for
the full tracker resolution already developed. A dedicated engineer at
Boston University has been identified. What remains to be decided is the
FPGA choice (will the XC2V6000 be available?) and whether to merge the
mother/daughter boards to improve robustness. The existing Run 2a DFEA
daughter board connectors have been a source of unreliability.
Consultation with Jamieson Olsen after the review indicates that the
connector failures are mostly metal fatigue failure in the cantilever
spring contacts causing the spring to break or no longer supply
sufficient contact force. An integrated design will likely lead to a
more mechanically robust solution, but may lead to a modest increase in
the layer count of the PC board. However, issues regarding the crate
power distribution, controller, and testing have been identified and are
discussed below.

The proponents of the L1CTT upgrade propose a 48V power solution in
place of the low voltage/high current solution of Run 2a. This
simplifies engineering and also improves the operation maintenance of
the L1CTT system since the proposal would allow all signal connections
to be made at the back of the crate. The risk is that the system may
induce EMI effects to other sensitive D0 electronics. The reviewers
recommend performing up-front engineering to ascertain that the design
of the 48V plant does not adversely affect other detector systems,
possibly performing an in-situ mock-up test. Various options regarding
the placement, shielding and wiring of the power plant need be
determined prior to finalizing the layout of boards or backplanes. It
may be advantageous to use a power system where each board has its own
converter on-board, but it may be better to have an array of power
converter units in an external box, with a twisted pair to each slot
position in the backplane.

The reviewers question the design choice made for the high-bandwidth
crate controller. The system requires only 2 such modules (plus spares),
yet engineering for a fully custom solution is proposed. The basis of
the design is a new VME controller under design by the CMS group at The
Ohio State University, which is proposed as a low-cost, radiation
tolerant design for the 60 peripheral crates of the CMS Endcap Muon
System. The design employs a gigabit Ethernet link (utilizing raw
Ethernet packets, no TCP/IP protocol), and a Xilinx Virtex-2 Pro FPGA
with RocketIO technology. However, since the L1CTT crates use a custom
read/write interface rather than VME, the OSU design must be modified.
The L1CTT proposal includes engineering to develop schematics, layout,
firmware, and testing for essentially a new board design. This in itself
indicates that the Project file should have explicitly a second,
pre-production, revision of the board built-in. The Linux software
drivers for the Ethernet interface developed by OSU can be re-used,
however, as can some of the core Xilinx firmware for the RocketIO
interface. We feel that the risk associated with developing a new
interface of this complexity, given the limited engineering and tight
schedule, is high. The OSU designers currently have a PC board in hand,
but the firmware and testing that must be done to validate the design
remains to be completed.

The reviewers are not convinced that a commercial solution (fully or
partially) could not be found to satisfy the bandwidth needs of the
L1CTT in order to justify the \$100K expense of the crate controller
project (and which is in danger of significant schedule slippage). Since
the backplane is under redesign as well, other interface and bus
technologies could be utilized. The software effort needed to integrate
a standard controller is likely to be less than that required to debug
software performing gigabit frame extraction.

As part of the DFEM motherboard development, the Boston group proposes
to develop a DSAT tester module that allows testing of the DFEM/DFEA
combination without a crate backplane and controller, which is to
mitigate the risk associated with schedule slippage of the latter. The
engineering resources at Boston are stated to be available and
independent of the DFEA/DFEM development. However, the reviewers
question the value of investing resources and time in developing this
modestly sophisticated tester board based around a Virtex-2 FPGA. Given
the large amount of RAM and logic resources in modern FPGAs,
particularly the XC2V6000 proposed for the DFEA, it is the opinion of
the reviewers that most of the test functionality could be built into
the DFEA/M module itself (e.g. add the PC and JTAG interface to the
board, and input/output spy FIFOs in the FPGAs). The specific objection
of not having the SCL link available off-site is also questionable. The
SCL is a rather simple protocol and easily simulated.

The proponents have placed a lot of value in being able to perform
in-situ tests of the upgraded L1CTT system during this year's shutdown.
Concerns about the schedule (see below), however, indicate that this
will be very difficult to meet. The reviewers thus suggest developing
contingency plans for testing outside of the collision hall using Monte
Carlo test vectors (or even data recorded by the current system). For a
digital system, this ought to be sufficient, provided that the necessary
test circuitry (input/output FIFOs) is built into the design. Objections
about timing skew at the inputs of the boards are met by simply
referring to the experience in the extant system, where skew has been
controlled by careful system design. In any case, there will be serious
conflicts for access to the collision hall during access times as other
upgrades to CFT and the calorimeter will be occurring simultaneously
with L1CTT work. This will inevitably increase any estimates of time
spent commissioning in this area.

The reviewers are of the opinion that the pinout of the connector on the
backplane is not optimal and will cause un-needed complexity, possibly
leading to signal quality issues and therefore a risk for cost overrun.
The design is, by definition, mirror-image in that each half of the
board performs logic upon one sector of the detector. The connector
pinout should also be mirror-image in that the cable order (see page 4,
Hazen talk) should be from top to bottom:

in1

in2

in3

in4

out1

out2

out4

out3

in8

in7

in6

in5

# Schedule

There is a risk for schedule slippage if the design decisions on the
power, FPGA choice, and board merger are not made soon. Given that the
48V option requires upfront engineering to check for any negative impact
on other D0 electronics, it is even likely that the schedule will slip
by at least the amount of time it takes to reach these decisions.

There is a concern that the identified engineering resources for the
backplane and controller development are insufficient to meet the stated
schedule. The backplane and controller tasks require \~1.3 FTE of an
FNAL electrical engineer over the next 6 months (April−September 2004,
assuming 1768 hours per FTE-year). The identified FNAL engineer for
these 2 projects is Jamieson Olsen, who when asked during the review, is
able to commit 0.5−0.75 FTE during this period. This indicates
significant slippage in the controller and backplane tasks, which when
combined with a second crate controller revision implies that the two
are not likely to be ready before mid-2005.

The LVDS splitter development task, scheduled to start May 2004, does
not yet have an identified FNAL engineer. There is about 1 month slack
before it is late for a summer shutdown test.

These schedule concerns indicate that making a September 1, 2004
deadline for installation of a partial (but complete slice) during this
year's shutdown is unlikely.

# Contingency

# 

The cost basis given for PC boards is predicated upon the
mother/daughter approach; the potential for moving to the integrated
design is justification for providing some excess contingency (on the
order of 15%) for the bare PC board fabrication line item.

The crate controller with the gigabit Ethernet link and Xilinx RocketIO
is a new design not based on the previous Run2a L1CTT controller, and
new technology for the identified engineer. The reviewers recommend
higher contingency, 30−40%, for the crate controller labor costs.

# Cost

# 

The L1CTT crate controller is a new design, but with only a single
development cycle listed in the Project file. The reviewers recommend
that a second, pre-production revision be put in explicitly.

Additional cost and schedule comments from Bill Freemen are attached
directly to the end of this file.

# 

# Milestones and Tracking

# 

Milestones should be added for three technical decisions that must be
made very soon (several weeks) in order to allow the project to proceed
on-time:

1.  Whether to design around a high-current/low-voltage power
    distribution as in Run2a, or to switch to a 48V power supply and
    employ DC-DC power converters on the boards. As stated earlier,
    testing should be done to ascertain that the 48V choice does not
    negatively impact other detector electronics.

2.  Whether or not to merge the DFEA daughterboard onto the motherboard.
    Merging avoids connector reliability issues, but prohibits future
    upgrades and will increase the complexity of the board to some
    degree.

3.  Choice of FPGA. Current target FPGA is the Xilinx Virtex XC2V6000,
    which is as of yet unavailable. Fall-back is the XC2V4000. The
    difference in chip resources between the XC2V6000 and XC2V4000 is
    stated to be 60% versus 80%, respectively. If not already done, a
    simulation of the Run 2b track equations in the XC2V4000, or a test
    using the existing DFEA daughterboard pre-production prototype,
    should be done to ascertain that a design around the smaller chip is
    feasible and without timing problems before committing to it.

Additional Cost and Schedule Comments

Reviewed by: Bill Freeman

# Findings

1)  The Project has incorporated the proposed changes into a draft MS
    Project resource-loaded schedule. The changes incorporate new WBS
    elements as well as changes to pre-existing WBS elements.

2)  Engineer, technician, computing professional, and physicist
    resources are assigned at the lowest task level.

3)  A portion of the CTT sub-project cost is funded via an existing
    NSF-MRI grant administered through Boston University. The remainder
    is funded through DOE funds. The Project intends to fund the
    increased scope of work through the allocation of existing DOE
    management reserve funds.

4)  The cost changes as presented to the committee are summarized in
    Table 1.

5)  The M&S costs associated with the new WBS elements are given in FY04
    dollars.

6)  The revised M&S costs associated with pre-existing WBS elements are
    given in FY04 dollars.

7)  Fermilab labor rates are based on FY02 dollars.

8)  The costs as presented do not include any indirect overheads applied
    by Fermilab. These are \~ 30% on FNAL labor, \~16 % on Fermilab M&S
    procurements, including FNAL P.O.s issued to universities not
    subject to the pass-through rate or 1.5% for those universities
    subject to the pass-through rate.

9)  Contingency fractions have not been assigned for many of the new WBS
    elements.

10) Cost account and work package entries have not been made for many
    WBS elements.

11) Three key decisions need to be made in the next few weeks for which
    there are no corresponding milestones in the schedule. These are:

    a.  A decision on the use of 48v crate power rather than 3.3 v power

    b.  A decision on whether a separate DFE daughterboard/motherboard
        combination will be retained or the two will be integrated into
        a single board.

    c.  Which FPGA will be used in the next revision

# Comments

1)  Attachment 1 details some specific missing or incorrect items found
    while reviewing the MS Project schedule file. This is not an
    exhaustive list. There may be other items needing
    addition/correction.

# Recommendations

1)  The proponents should carefully review the resource profiles arising
    from the RLCS to insure that key personnel are not overcommitted.
    This is especially important in view of the aggressive schedule
    required to be ready for the proposed prototype testing phase
    following the late summer \'04 accelerator shutdown. Adjust the
    schedule accordingly.

2)  The proponents should make comparisons between cost numbers that are
    in the same units, i.e. FY02 to FY02 dollars or FY04 to FY04
    dollars, to determine the amount of management reserve actually
    needed.

3)  Indirect costs should be included in the determination of the amount
    of management reserve needed to fund the CTT improvements.

4)  The project management should determine how it wishes to
    handle/assign contingency on the new WBS elements. The trigger
    project management should review the contingency that remains as a
    fraction of the ETC to confirm that the remaining contingency is
    adequate.

5)  Add milestones to the MS Project schedule for the three decisions
    mentioned in Finding 10.

6)  The proponents should thoroughly review/comb the MS Project file
    with the DZero scheduler to enter information that is currently
    missing or out-of-date on various tasks, and correct other small
    errors. This includes entries in the \"Responsible Institution\" and
    \"Funding Source\" fields, the \"Cost Account\", \"Work Package\",
    and \"PMT\" fields, and the \"Notes\" field that contains the WBS
    Dictionary and Basis of Estimate information. BOE information should
    be made consistent with the revised cost and labor numbers.

7)  Backup material justifying new cost estimates should be collected
    and collated for easy reference.

8)  Once the change request is approved, the appropriate set of tasks
    should be (re)baselined in the MS Project file and the new baseline
    information, cost accounts, and work packages should be added to the
    project\'s COBRA program.

**TABLE 1**

**Cost Changes Presented to the Committee at WBS Level 4**

![](media/image1.png){width="2.192361111111111in" height="8.75in"}

**ATTACHMENT 1**

[Items Needing Correction in MS Project file]{.underline}

1.  Several of the new CTT Milestone tasks currently have durations of
    1d. These should be set to zero days. These are:

> 1.2.3.8.3 interface specs (hardware+timing) finished
>
> 1.2.3.8.3 interface specs (hardware+timing) finished
>
> 1.2.3.8.10 ready for production
>
> 1.2.3.8.12 first BP received
>
> 1.2.3.8.17 delivered to Boston
>
> 1.2.3.9.11 CC ready
>
> 1.2.3.10.3 specs done

5.  ready for platform test

```{=html}
<!-- -->
```
2.  Delete blank task (ID 322)

3.  Tasks 204, 205, 228, 241-287, 306-323; Most tasks in these ranges
    are missing institution and funding source entries, as well as cost
    account, work package and PMT entries. Task 288-305 have entries but
    the cost account and work packages are incorrect.

4.  Tasks 245, 248, 250, 262, 321 are missing resource assignments.

5.  Task 276 is called \"hardware procurement\" however there is no
    associated M&S non-labor dollar amount entered for this task? Should
    there be?

6.  Task 209 (1.2.3.6.1.5 Test) has both InKind and MandSPass resources
    assigned. This will need to be split into two tasks, if it is truly
    to be funded both by equipment and MRI funds.

7.  Task 313 has \$7000 budgeted as MandS labor for BU technician
    support however only 11 hours of Tech time is assigned to the task.
    That\'s a pretty expensive tech!

8.  Task 314 has a task type of \"Fixed Work\" (and is effort driven) --
    it should be fixed duration (and not effort driven) and
    hours/duration adjusted to reflect the anticipated budget need.
    (\$18,000?); Task 314 also has a \"deadline\" date of 4/3/04 that
    should be removed.

9.  Suggest renumbering WBS 1.2.3.16 subtasks to begin at 1.2.3.16.1
    rather than 1.2.3.16.2, since this part has not yet been baselined.

10. Task 221-222, 230; Task 222 has same description as 221 and 230 but
    only has ETU resource assigned and explicitly has INKIND\[0\]
    assigned. Also the three tasks have different predecessors and
    different start dates. This should be checked/corrected as
    necessary.

11. There are a number of tasks where the university labor hours are not
    consistent with the indicated M&S labor cost. Several of these are
    on tasks that are funded out of the MRI so in some sense this
    inconsistency doesn\'t matter as far as the use of EQU contingency
    goes. However it may matter in terms of the accuracy of the
    estimated labor needs and hence it may impact the schedule. In
    several cases it seems that a labor rate different from that stated
    in the notes was assumed. For example, on task 223 there is a
    reference to \$40 per hour + 61.5% university overhead in the notes
    which does not correspond to the \$22.5 per hour effective hourly
    rate one gets by dividing the budgeted inkind dollar amount of \$
    1800 by the 80 hours of budgeted labor ( 40 tech + 40 engineer).

12. Many of the notes that describe the WBS dictionary and BOE are
    either missing or have not been revised to reflect the proposed
    baseline changes. These should be adjusted/corrected, where
    appropriate.