# Presentation: 351225

**Installation**
** ****of the **
**D0 RunIIb Silicon & Trigger Upgrades**

**R. P. Smith for**
**The D0 Collaboration**

**Notes:**
Let me tell you some things about the planning that is underway for the Installation and Technical commissioning of the upgrade, expected to occur in 2005.

In my presentation I hope to acquaint you with our thinking regarding the management plan, schedule, and anticipated costs for these activities. 

##  Just What’s Involved Here?

- L0 Installation
- Longest Duration of all Upgrade Processes: defines Overall Tevatron Off-time
- Estimated 14 weeks including contingency
- Requires Break in Tevatron Vacuum at D0
- Requires Full Opening of Detector, Significant Collision Hall access
- L1 Central Track Trigger (“CTT”)
- Requires few weeks Collision Hall Access with intermittent short returns thereafter until re-cabling debugged and verified
- L1 Cal Trig Installation
- Requires no Collision Hall Access
- Complete Disruption of Experiment’s Trigger since entire Cal Trigger electronics removed and replaced
- L1 Cal Track Match, L2Beta,  L2STT
- No Collision Hall access, non-disruptive
- Interleave above with Detector maintenance and repair
- Power supplies, individual channel recovery, safety systems

**Notes:**
Turning now to a closer look at the Installation schedule itself, we find that the beam shutdown time required to install the silicon is approximately seven weeks.  

Furthermore, we can use the schedule to enable us to predict how far in advance of the completion of the silicon we prefer the beam shutdown to occur.  This lead is approximately 3 weeks.

The schedule shows us that the trigger elements require much less access to the collision hall – I will come back to that issue – and that personnel can leave the collision hall as soon as the electronic checkout of the silicon is completed.

The schedule indicates that we prefer to leave the detector open for a few more weeks to facilitate short accesses to the CH consistent with Tevatron turn-on.   Prior to full physics commissioning we would take the time for final survey and close-up of the detector.

## Actually Have a “Running Start”

- Upgrade Installation Tasks Already Achieved:
- L0 Silicon – L0 clearance survey; extensive practice with installation tooling; adapter card and 4-Hybrid SVX4 string operating in Gap since 2004 shutdown; N&S mini fuse panels, new Wiener PS and Interlock chassis, Temperature monitoring IO installed
- L1CTT - Parallel “slice” running since 2004 shutdown: crate, controller, few DFEB’s, installed and operating in detector; Doublet equations on real data exactly the same; Singlet equations under study
- L1 Cal – Operating with Slice of Real Calorimeter BLS Data on Sidewalk
- L1 CalTrack – L1 triggers (L1 Muon mod + L1CTT + L1Cal) thru trigger and manager crates to L3; data available for unpacker tests; L1CTT transition and MTCxx cables terminated
- L2 Beta – Instance of new processor running in L2 for some time
- Muon Pipeline depth (L1 Latency) -- Instance of PDT mod now operating in detector
- Online upgrades completed

**Notes:**
The previous speakers have already identified certain “infrastructure” tasks for their portion of the upgrade that are planned for the 2004 shutdown of the Tevatron.  These tasks typically take advantage of full access to the detector or, in the case of the Online, permissible short downtimes of the D0 DAQ.

Detail regarding these activities has been incorporated in the Installation schedule we have prepared to help manage the Upgrade installation.

## Overall Strategy

- Minimizes overall Luminosity Cost by installing Upgrades in parallel, as rapidly and safely as possible, to ensure integrity of overall Tevatron program at Fermilab
- Depends on cooperative alignment of scheduled work by BD crews for safe, error-free work with TeV beampipe components at D0
- Depends on modest help from AMG (4 one-day, 1 two-day job*) at scheduled times during shutdown
- Depends on critical skills of trained detector specialists during L0 work

*** Unless done before-hand**

## Overall Strategy, cont’d

- Meanwhile, during the next few months
- Incorporate L1Cal Trig into regular Global Runs in Control Room after ~mid-November
- Plan to substantially complete “adiabatic” installation of B&C Layer PDT & SFE mods for latency change
- Incorporate four SVX4 hybrid strings into readout to fully exercise L0 readout chain
- Push maturation of Straw-Man Trigger List and associated tools to well-understood “V15” Trigger List for running at and beyond 150E30 when hardware installation finished
- Continue L1Cal and L1CTT “slice-test” maturation
- Rehearse L0 Installation once again in Lab 3
- Continue with L2Beta software maturation
- Incorporate Cal Trk Match into physics triggers

## Some Planning Nuts & Bolts

- L0 Installation Schedule Duration: 14 weeks/69 work days beginning March 1
- Careful planning
- (L0) installation fixture development
-  Mock-up of critical L0 steps
- Develop special tooling to reduce opening/closing of EC’s/EF’s as Tevatron beampipe is disconnected/reconnected and L0 tooling is installed
- Incorporate valuable experience from 2004 Shutdown  when special L0 clearance measurements were made
- Create contingency by moving lower-risk tasks to 2nd shift
- Understand Overall Manpower Requirements
- From FNAL D0 Group, Universities
- Specific personnel from Laboratory

**Notes:**
The previous speakers have already identified certain “infrastructure” tasks for their portion of the upgrade that are planned for the 2004 shutdown of the Tevatron.  These tasks typically take advantage of full access to the detector or, in the case of the Online, permissible short downtimes of the D0 DAQ.

Detail regarding these activities has been incorporated in the Installation schedule we have prepared to help manage the Upgrade installation.

## L0 Installation

- Plan with fully loaded MPP
- Assign experienced personnel with special “body awareness”, (and where necessary) long arms, for close work in tightly constrained space,  who must install 2m long multi-million dollar one-of-a-kind object in opening with ~ 1mm clearance
- (Show L0 Technical Commissioning only because it must be completed before close-up)
- Five-day work week, Lab holiday schedule
- Manage parallel tasks to optimize use of N and S Gaps, minimize overall duration, without generating additional risk

## Silicon Installation Milestone Overview

| ID | WBS | TASK NAME | START-DATE | FINISH-DATE | DURATION |
| --- | --- | --- | --- | --- | --- |
| 2 | 1.5.1 | Beginning of RunIIb Tevatron Shutdown | 3/01/06 | 3/01/06 | 0 w |
| 28 | 1.5.2.2.10 | Detector Open, Ready for Access | 3/09/06 | 3/09/06 | 0 w |
| 34 | 1.5.2.3.5 | RunIIa Be Pipe Disconnected | 3/13/06 | 3/13/06 | 0 w |
| 45 | 1.5.2.4.10 | H Disks Removed | 3/20/06 | 3/20/06 | 0 w |
| 53 | 1.5.2.5.7 | RunIIa Be Beampipe Removed | 3/29/06 | 3/29/06 | 0 w |
| 72 | 1.5.2.8.5 | L0 Tooling and Mounts Ready | 4/19/06 | 4/19/06 | 0 w |
| 76 | 1.5.2.9.3 | L0 Installed | 4/21/06 | 4/21/06 | 0 w |
| 85 | 1.5.2.10.8 | RunIIb Be Pipe Connected, L0 Cabled | 5/04/06 | 5/04/06 | 0 w |
| 88 | 1.5.2.11.2 | H Disks Installed | 5/10/06 | 5/10/06 | 0 w |
| 93 | 1.5.2.12.4 | Silicon Cold and Ready for Technical Commissioning | 5/15/06 | 5/15/06 | 0 w |
| 97 | 1.5.2.13.3 | Complete Technical Commissioning of Silicon | 5/19/06 | 5/19/06 | 0 w |
| 113 | 1.5.2.15.9 | Detector Closed for Tevatron Resumption | 6/06/06 | 6/06/06 | 0 w |

**Tevatron Downtime**
**14 weeks (69 working days)**

**Milestones Shown are from Single Shift Schedule**

**Notes:**
Copying from the MS Project schedule, the milestones for these activities are as shown.  A Tevatron shutdown of at least 7 weeks is required, not to occur more than three weeks prior to shipment of Layer Zero from SiDet.

## L1 Cal Installation Tasks

**L1 Cal Ready for Technical Commissioning **
** ****~ 2.5 weeks before beam Resumes**

| 
ID | 
WBS | 
TASK NAME | 
START-DATE | 
FINISH-DATE | 
DURATION |
| --- | --- | --- | --- | --- | --- |
| 2 | 1.5.1 | Beginning of RunIIb Tevatron Shutdown  | 3/01/06 | 3/01/06 | 0 w |
| 134 | 1.5.3.2.1.2 | Final Cal Noise Studies | 3/10/06 | 3/13/06 | 0.2 w |
| 135 | 1.5.3.2.1.3 | Decable BLS cables from Trigger Crates | 3/13/06 | 3/20/06 | 1 w |
| 136 | 1.5.3.2.1.4 | Depopulate & Remove Run I Trigger Crates | 3/20/06 | 4/03/06 | 2 w |
| 137 | 1.5.3.2.1.5 | Install Rack Infrastructure in Active (ADF) racks, Install Patch Panels in Passive Racks | 4/03/06 | 4/24/06 | 3 w |
| 138 | 1.5.3.2.1.6 | Reconnect BLS cables | 4/10/06 | 4/24/06 | 2 w |
| 139 | 1.5.3.2.1.7 | Install ADF, TAB/GAB, Readout & Controller Crates | 4/24/06 | 5/01/06 | 1 w |
| 140 | 1.5.3.2.1.8 | Connect Pleated Foil Cables | 5/01/06 | 5/08/06 | 1 w |
| 141 | 1.5.3.2.1.9 | Connect ADF – TAB/GAB  Cables | 5/08/06 | 5/15/056 | 1 w |
| 142 | 1.5.3.2.1.10 | Connect L1/L2, TFW, SCL, L1CalTrkMat Cables | 5/15/06 | 5/17/06 | 0.4 w |
| 143 | 1.5.7.2.1.11 | L1 Cal Ready for Technical Commissioning | 5/17/06 | 5/17/06 | 0 w |

**Milestones Shown are from Single Shift Schedule**

**Notes:**
Copying from the MS Project schedule, the milestones for these activities are as shown.  A Tevatron shutdown of at least 7 weeks is required, not to occur more than three weeks prior to shipment of Layer Zero from SiDet.

## Contingency Analysis

- Can Categorize tasks according to risk to learn where contingency might be earned
- Some tasks “incompressible” – Be hazard analysis, SNEG activation, glue curing, manipulation of detector elements
- Some tasks lower risk, eminently suited for 2nd shift work: routine detector reconfiguration, scaffolding setup/removal, setting up beam tube purge lines and fixtures
- Some tasks incompatible with 2nd shift work – require great care, precision, supervision: TeV Be pipe extraction, L0 Mount attachment, L0 Insertion, RunIIb Be pipe installation
- Prudent to Schedule, and Manage to, “2nd Shift” Tasks to Earn Contingency

## Second Shift Tasks

- Second Shift work earns 7 working days contingency

| WBS | Task | Start | Duration* |
| --- | --- | --- | --- |
| 1.5.2.2.2 | Open Muon Shields, EF’s | 3/01/06 | 0.1 w |
| 1.5.2.2.4 | Remove FPD Vetoes | 3/2/06 | 0.2 w |
| 1.5.2.2.7 | Open EF’s, CF’s, EC’s | 3/3/06 | 0.2 w |
| 1.5.2.3.1 | Prep for EC/Be Disconnect; Rad Survey | 3/6/06 | 0.1 w |
| 1.5.2.3.3 | Cut EC Pipe ends; Record Temps | 3/7/06 | 0.1 w |
| 1.5.2.4.2 | Stop SMT Coolant, drain, purge | 3/8/06 | 0.1 w |
| 1.5.2.5.3 | Close ECN,EFN, Make Platform at EFN | 3/20/06 | 0.2 w |
| 1.5.2.5.5 | Remove Be Pipe from CH | 3/21/06 | 0.1 w |
| 1.5.2.7.1 | Remove N Scaffold, Open EFN, ECN, etc. | 3/29/06 | 0.2 w |
| 1.5.2.10.6 | Weld EC Beampipe Stubs | 4/26/06 | 0.2 w |
| 1.5.2.14.1 | Remove Gap Hardware, Close EC’s, EF’s, | 5/15/06 | 0.2 w |
| 1.5.2.15.3 | Close CF’s, EF’s | 5/22/06 | 0.2 w |
| 1.5.2.15.5 | Extend Muon Shields, Close Clamshells | 5/23/06 | 0.2 w |
| 1,5,2,15,7 | Test Magnet Power Supplies | 5/24/06 | 0.6 w |

*** Some Tasks were already in parallel with others**

## More Contingency Analysis

- Saturdays represent contingent work days – potentially 13 days , but many task require specialized skills, are physically challenging, so more reasonable harvest: ~ half of Saturdays
- Technical commissioning of Silicon allowed 4 days; generous by one or two, given physicists can also work at night
- Power supply testing at end allowed 3 shifts; two available for unforseen tasks
- Summary of Contingency:
- Night Shifts – 7 days
- Saturdays – 6+ days
- Silicon Commissioning and Power Supply Testing – 3 days
- Total: 16+ days
- 23+% Contingency not seen as excessive for project of this complexity

## How Might We Spend Contingency?

- Be Analysis requires unexpected handling measures
- Beampipe leak checking after reconnection
- A leaking joint could require detector reconfiguration
- Calorimeter beampipe temperatures lower than anticipated
- Other unanticipated complications

## Identified Additional Manpower Needs

- Assuming university teams and lab resources currently available to D0 remain available, the identified supplemental manpower requirements are as follows:
- L0 Installation Expert Engineers: Youri Orlov, Joe Howell, Dan Olis, (Mike Hrycyk)
- L0 Installation Expert Instrumentalist: Mike Roman
- L0 Installation Expert Technicians: Dave Butler, Ken Schultz, Bert Gonzalez, Ron Davis
- PDT Expert: Sten Hansen, (Boris Baldin)
- Four Electrical Techs for PDT mods ( 2 weeks)
- Four Mechanical Techs (duration of shutdown)
- Welder (1 day)
- CTT Mixer Engineer: Stefano Rapisarda
- AMG teams (2 3-man crews, 3 2-man crews)
- AD beampipe specialists (4 days)
- Fermilab engineer for L1Cal integration/technical commissioning
- Short term visitor support for L1Cal Track Match
- The Lab is aware of this list and remains supportive – we anticipate the needs will continue to be addressed

**Notes:**
Following technical commissioning, which demonstrates downloading of crates, crate readout and control,  proper mapping of electronic to detector channels, proper timing-in with the TFW and beam crossings, etc. it is then necessary to understand beam-on L1-L2-L3 accept rates, trigger turn-on and efficiency, noise rejection, etc., to ensure that full physics capability of the detector has been restored. 
To understand how the return of the detector to physics will occur, the spokespersons have convened a standing committee on prepare a timeline and resource/cost table which quantifies this process.

## Installation Effort

**Workweek = Standard 5 days per week**

**Notes:**
During the actual installation period the demand for technicians is high.  Very little two-shift work is called for in the schedule – that during the opening of the detector being an important exception.

Peak weekly FTE’s (that is effort binned in bins of 1 week duration) is plotted against calendar weeks, with the beam-off period explicitly indicated. 

## Installation Effort 

**Workweek = Standard 5 days per week**

**Notes:**
During the installation period the demand for engineers is also substantial.  None of the work is two-shift, and significant effort will be provided by the universities involved in the trigger upgrades.

Again, peak weekly FTE’s (that is effort binned in bins of 1 week duration) is plotted against calendar weeks, with the beam-off period explicitly indicated.  The engineers continue to assist during the out weeks as technical commissioning proceeds.

## Installation Effort 

**Notes:**
During the installation period the demand for physicists, both from Fermilab and the universities,  is also substantial.  During some of the commissioning tasks, physicists as a group will clearly be at DAB during at least two shifts per day.

Again, peak weekly FTE’s (that is effort binned in bins of 1 week duration) is plotted against calendar weeks, with the beam-off period explicitly indicated.  Physicists remain deeply committed at DAB during the out weeks as technical commissioning proceeds.

## Detector Gap Occupancy 

**Per Gap Limit: 2**

**Notes:**
It was thought prudent to make a “reality check” on how the schedule populates busy regions of the detector.  To this end “gap” and “cathedral” occupancy resources were assigned (apologizing for the MSProject-speak) to each task where people were to be working in these areas of the detector.

The schudules show that the gaps are not overloaded (when two shifts at times are considered), nor is the cathedral area.   For a gap, two people maximum can work at one time, leading to 4 persons peak on a 1-shift basis, or 8 persons for two shifts per day. 

## Conclusions

- Conservative, Informed Evaluation of Duration of Installation Tasks
- MPP Used to Evaluate Total Duration, Effort
- Contingency earned via select 2nd shift work, some Saturdays
- 14 week shutdown schedule includes ~25% schedule contingency
- Assuming that the manpower resources outlined within this presentation are available, we expect to be able to implement this plan successfully
- Success depends on significant dedicated effort from collaboration and personnel supporting D0, in addition to itemized additional non-D0 personnel, close cooperation of AD, timely assistance of AMG

**Notes:**
The collaboration understands the need for careful planning to ensure that the upgrades are installed and detector returned to physics with as little lost luminosity as possible.

D0 has put in place a management team, a planning schedule, and a special standing committee to provide detailed predictions about how physics commissioning must be accomplished.

Finally, D0 prefers that a 2005 shutdown of the Tevatron be aligned with the completion of the Layer Zero silicon, or later.