# Presentation: 238089

## Joint CDF/D0/CD  Computing Project OverviewRuth Pordes, Jun 8th ‘99

- Congratulations to Babar, KTeV and SDSS on data taking ...
- 	Summary of Review Agenda
- 	Other Milestones in the last six months
- 	Response to Jan ‘99 Reviewers Recommendations
- 	Integration and Operations
- 	Mass Storage Hardware
- 	Stuff

## Summary of Agenda

- Projects reaching end of Development Phase - Patty Mcbride
- Data Access and Databases
- 	Data Access: D0 - Vicky White
-   Storage management: Enstore - Jon Bakken
-   Data Access: CDF  - Stephan Lammel
-   Support Databases - Ruth Pordes
- Evaluations and Decisions on Serial Media- Dan Amidei
- Hardware Procurements
- 	Physics Analysis Hardware - Dave Fagan
-   Network Hardware and RIP - Phil Demar
-   Production Hardware  - Lisa Giachetti
- ROOT and IDL - Physics Analysis Software - Jeff Kallenbach and Philippe Canal
- Production Management - in 3 parts - Marilyn Schweitzer, Tony Chen and Heidi Schellman.
- Plans for Operations - Dane Skow and Matthias Kasemann

## Milestones in the last six months

- CDF and D0 Robots on the floor at FCC. D0 Robot in use for Monte Carlo Challenge; CDF Robot in last stages of installation**.**
- FARMS prototypes and FBS software put to good use.
- Joint Project Deliverables met towards D0 Monte Carlo Challenge.
- Joint Project Deliverables met towards CDF May Milestone.
- Physics Analysis Hardware purchase order (nearly) placed.
- ROOT Collaboration Established.
- Projects ready to transition to Maintenance and Operations:  Mass Storage Hardware, ZOOM, Configuration Management, Simulation and Visualization.

## Recommendation 1:

- 1. D0 and CDF both using ROOT is very good. CD should aggressively move to support ROOT. This should be done as a specific effort in concert with the ROOT developers, not as part of any of the existing common projects.
-      **Response:**
- There has been much progress in this area. A successful workshop for US HENP Experiments was held in March which was attended by the whole ROOT core development team. A list of required enhancements for Run II use was made and assignment of responsibilities between the contributors - now including Fermilab development and support people.
- Reported in detail in the Physics Analysis Software Talk

## Slide 5

## Recommendation 2:

- ZOOM should not add any new projects. It should be in a maintenance phase.
-      **Response:**
- The status of Zoom will be presented.
- The original deliverables of this project are in maintenance mode.
- The collaborations have requested the addition of a few utilities of common interest and  worth.
- Collaboration with CLHEP is expanding.
- (Experimenter heard to say this is very good quality code even if not everyone uses it).

## Zoom Release 3rd June - releases every month or so

- Main Utility Change: Compatible with the **--thread_safe** option in KAI
- Platforms checked: KAI on d02ka  with D0 environment; KAI on cdfsga with CDF environment; KAI on cdfpca with CDF environment; NT under MSVC++ 6.0
**HepTuple:**  Fixed Root manager's treatment of column arrays;  Fixed Root manager's director management; Cleanup of inappropriate CXXCFLAGS in makefile
-  Platforms checked:   KAI on d02ka  with D0 environment; KAI on cdfsga with CDF environment; KAI on RedHat 5.0 (cdfpca) with CDF environment; NT with Microsoft C++ 5.0 with D0 environment; egcs 1.1.1 on fnpat1; Warning: In a future release the HBOOK part of the HepTuple; library WILL be separated from the manager independent part. Software using the hbook part of HepTuple will need toadd -lHBookHepTuple to their link lines.
**CLHEP:** Reflects Official CLHEP repository versiions; Minor tweaks to let packages compile on a broad  variety of platforms without issuing warnings; Strengthened validation for Gaussian distribution. Platforms checked:   KAI on dochb  with D0 environment;  KAI on cdfsga with CDF environment; NT with Microsoft C++ 5.0 with D0 environment;  KAI on RedHat 5.0 (cdfpca) with CDF environment; Various platforms at CERN, including DEC, SUN...
- Minor changes to:  Exceptions;   ErrorLogger;   PhysicsVectors

## Recommendation 3:

- CD should support visualization products such as HEPVIS, Openinventor, ROOT and the experiments should use them.
-      **Response:**
- The Visualization Joint Project has done work in the deployment and support of HEPVIS and Openinventor (ROOT is under the umbrella of the Physics Analysis Software Project).
- A remaining issue is the ongoing support of Openinventor for new versions of the KAI compiler and Linux and whether we should purchase a Site Licence.
- Fermilab active in HEPVIS Collaboration and will attend September HepVis ‘99.
-      **D0 Response:**  Dzero is making use of OpenInventor, HEPVIS and ROOT.

## Recommendation 4:

- The C++ working group should continue to resolve the practical aspects of using these new technologies, with an emphasis on immediate needs.
-      **Response:**  Yes.
- The work of the C++ working group has continued with lively participation from both collaborations and the "C++ gurus".
- The group  has been pragmatic in its evaluation of compilers other than the KAI compiler adopted as standard for Run II.
- Attendance continues from time to time from SGI or/and Kai representatives.
- Kai Licences maintain their popularity....

## Recommendation 5

- Adding support to the Common Database Support Group is important and necessary. CDF and D0 should take the lead in defining the common infrastructure for the group to provide. The CD proposal to get a database consultant to teach techniques is a fine idea and the experiments should take advantage of it.
-   **Response:**
- This has happened in part with the hiring of a consultant for 2 weeks to help with hot backups and replication
- Areas Needing Attention:
-  Large Database Architecture and Design
-  Reviews of the Database Schema before data taking to address scalability and performance issues
-  Monitoring and Administration tools to help ensure robust, performant and high availability.
- Oracle will visit 21/22 June. Hope this is first of several visits.
**D0 Response:** Database support and work on common infrastructure is ongoing.

## Recommendation 6:

- No CD effort should go into an NT based analysis station.
**     ****Response:**
- No CD effort has gone into this project.
-      **D0 Response**: Dzero is requesting NT support for its Level III trigger, not for analysis stations. Some of this support will now be provided by summer Guest Scientist hired with D0 equipment moneys.
- Use of Linux is increasing rapidly at the Laboratory.
- Containement approach to NT for Run II still working. Still some small support needed for CDF Online as well as D0 Level 3.

## Recommendation 7:

- The Serial Media Working Group is an excellent collaborative effort and should follow the situation with the next generation tape drives appropriately.
-      **Response:**
- This will be elucidated in the talk on Serial Media.
- But no Baby So no Cigars.

## Recommendation 8:

- The committee recommends explicitly to integrate SAM/ENSTORE pieces into the forthcoming D0 milestone Monte Carlo Challenge in Feb-Mar '99 to test the interface and integration of the data access and data processing.
-      **D0 Response:**
- We have successfully used SAM/ENSTORE to analyze the Monte Carlo Challenge files.

## Recommendation 9 and 10:

- D0/CD should develop realistic user access patterns for analysis to demonstrate the scalability of the SAM/ENSTORE system.
-  **D0 Response:** We are continuing to build on the original work done a year ago, reported as the "seasons of SAM", to discuss realistic user access patterns for analysis to demonstrate the scalability of SAM/ENSTORE. We have not yet used the system extensively enough to have proven scalability to Run II rates. This work will start in the summer with the availability of sufficient hardware and the first production versions of the SAM and Enstore products.
**10.** We recommend careful planning well ahead of the start of the run for D0-specific home directory and program file servers.
-      **D0 Response:** The planning for a D0-specific home directory and program file servers is well underway. We intend to use our central analysis server or another O2000 machine for this.

## Recommendation 11 and 12:

**11.** The committee recommends that tests be carried out on a larger scale once the DIM design has been refined and the prototype advanced. A future prototype should include multiple tape access and tape writing (as well as reading) to the extent that drives are available by summer of 1999.
-      **CDF Response:** We agree with this recommendation. We expect the suggested additions to the prototype system to be implemented this summer. This is about 2 months later than our original WBS schedule.
-  **12**.We agree with CDF’s plans for the DIM development to match the baseline analysis system by the end of 1999. Extensions to include multiple SMPs should not slow down the baseline development.
-      **CDF Response:** We have deferred design work on the mulitple SMP extensions to the DIM until a baseline product is in place.

## Recommendation 13:

- The committee recommends to build the Central Analysis Server based on Fiber Channel connected disk drives instead of SCSI.
**  ****Response:**
- This was included in the Request for Procurements. Work is continuing on the evaluation and investigation of Fibrechannel disk.
- There is a lack of manpower in this area. An ad-hoc group has been assigned to do work - initially to address the issues of disk throughput on the D0 Online Alpha.
- Large Disks should be purchasable easily... it is beginning to feel like the ‘80s again...

## Recommendation 14:

- The committee strongly recommends close collaboration between CDF, D0 and the networking group to meet the networking requirements and to implement them in a timely fashion.
**Response:**
- This will be reported in the Networking presentation - you will be given a quiz on all the router and interface names at the end of the review...
-  A Network Task Force meets to provide a forum for discussion and review.
-      **CDF Response:** We are currently working with the network group as recommended by the committee.
-      **D0 Response:** Dzero has been working with the CD networking group on our networking plans. Vertical slice tests will ensure that the components perform as required.

## Recommendation 15:

- We recommend that a benchmark program constructed from the reconstruction packages be developed and used to compare the performance of the systems under consideration.
**Response:** We investigated and used the Spec95 suite as the benchmark for the FY99 Physics Analysis Hardware procurements.
- Please note TINY took 3 minutes to compile and run. SpecInt95 takes 6 hours...
- Work was done with the D0 TRF++ C++ tracking code which showed linear scaling with SpecInt95.
- Compiler options for SpecInt 95 and SpecFp 95 were chosen that reflect as far as possible the compiler options being used for the collaboration reconstruction code.
- Subsequent to this work the Sun Spec Organization representative requested Fermilab input to the constraints to be published for the Spec2000 suite. This was done.
-      **D0 Response:** We tested a version of our standard track finding package, TRF++, against standard benchmarking programs on various  platforms. This information gives us confidence that the procurement of the central analysis servers was done using reasonable benchmarks.

## Recommendation 16:

- We recommend that the hardware procurement schedule should proceed aggressively.
**     ****Response:**
- An award is imminent for the FY99 Physics Analysis Hardware purchase;
- An RFP is imminent for the FY99 purchase of Production Hardware.
- We are behind in the purchase of Serial Media because of lack of vendor delivery of new technology... you will hear all about this.
- We are on track with our Spending Plan.

## Recommendation 17:

- We encourage CDF, D0 and CD to expand the effort on project management by including real names and to update the WBS on a regular basis.We reiterate our recommendation to get professional help in this area.
**Response:** We decided early on not to put actual names on the WBS. For status reports - as you will see in talks in the upcoming review -  names are listed in order to provide realistic presentation of effort. There have been at least 2 updates of the overall WBS since the last review.
- Some Project Leaders find the WBS format and structure difficult to work with. Rather than have inconsistent information in different formats, we have not been strict about requiring regular WBS updates.
- There has been a spectrum in the diligence with which Project Leaders have been able to keep their WBS up to date. Storage Management,  networking, ZOOM and Production Management have been most proactive. SAM maintains a WBS outside of the standard MSProject  infrastructure and posts their WBS regularly. More professional help has not been assigned to date.
- It has proven difficult to use the WBS mechanism as effectively as hoped to present the realities of frequent changes in software tasks and milestones, or in presenting the ongoing support and maintenance tasks that take significant effort at all phases of a software project.
**D0 Response:** Microsoft Project updates continue.
**Joels Response:** I told you so.

## Recommendation 18:

- We suggest to consider a reallocation of material funds to finance manpower for high priority projects, if other sources cannot be found.
**CDF Response:** We have reallocated hardware money to finance a CP to work on the CDF data handling system
**D0 Response:** We have reallocated hardware money to finance 50% of a CP to work on the SAM project and on a CP for 4 months to work on NT configuration management infrastructure for the Level III trigger.
**My Response:** Can we do this again next year?

## C++ Gurus - Now there are two  - Continue their Valuable Work - Now they have a web page -

## Milestones Still to be Met

**Oct ‘99 - D0 Mock Data Challenge**
**Oct  ‘99 - CDF Mock Data Challenge**
**	****	****BEAM**

- Quote from Ferminews:
- The Science of Run II: The Main Injector will fire up the Tevatron for making new
- discoveries. Run II of the Tevatron offers Big Science that even a banker can love:
- the promise of making spectacular discoveries without making a spectacle of the budget.

## Integration

- A lot of attention and work is being paid to this area. This is a measure of Real Progress.
- Integration involves a lot of people from different groups with specific expertise and responsibilities. Means more meetings (gatherings) than usual. Looking to  identify people with specific Integration Planning and Execution Responsibility.
- Integration of the Hardware Includes: Production Farms, Database Servers, Robots, Data Handling Systems, Tape Drives, Networks, Unit Tests, Test Systems.
- Need: System Monitors and Administration, Database Monitors and Tuning, Robot Statistics and Robustness, Network Monitors
- Integration of the Software Includes: Data Handing, Databases, Analysis Frameworks, Test Data
- Need: Monitors and error reporters; working interfaces; fast turnaround to small problems;

## Operations

- We have been working intensely in the last 4 weeks on the Operations Needs Assessment Document (to parallel the DAMNAG and the SNAG).
- Dane will report on the status and the assessment.
- Experience is once you start to look at the details you start the count.
- Operations always starts earlier than you think (or Development goes later than you think??)
- Operations must cover maintenance and upgrade of hardware and software.
- More Later.

## Mass Storage Hardware Status

***Mass Storage Hardware Status***

- D0 Robot in use by SAM and Enstore.
- CDF Robot almost completely installed.
- Fire Supression Systems are needed. They will require a couple of weeks down time for each Robot.
- Seems like ADIC repair man has to come fairly frequently still.
- Variety of drives being used in the robot.
- Joint Project moving to Maintenance Phase.
- Operations will be handled by CD operations departments and hardware calls through  the regular Service Call mechanisms.
- Thank Mark Leininger for his quality work as Project Leader and relieve him of his duties - including reporting at the Tuesday Run II meetings every week.

## Stuff

- The CDF D0 CD  Run II Joint Offline Project continues with lots of progress and hard work. People attend meetings together and talk:
- Tuesday: SAM, Enstore, Support Databases, Physics Analysis Software,  Steering Committee, Run II Computing
- Wednesday: ZOOM/CMGT/C++ working group (each week is something different). CDF Data Handling(? It got moved?)
- Friday: Physics Analysis Hardware; Run 2 Farms; Next Generation Operations/Network Task Force.

## Stuff

- Maillists see lots of activity from all points:
- CDF_UPGRADE                CDFDB-USERS                         CDFDB-SUPPORT
-  CDF-MOD-DB                  CPLUSPLUS-SUPPORT           CPLUSPLUS-WG
-  CVS-SUPPORT                 D0DB-USERS                            D0DB-SUPPORT
-   ENSTORE                        GDESIGNER-SUPPOR T          JAVA-EVALUATION
-   KAI-SUPPORT               LINUX-USERS                            PASREC
-   RUN2FARMS                 RUN2MASS                                 RUN2PAH
-  RUNIICOMPUTING       SAM_DESIGN                             SOFTRELTOOLS
-   SWENG                           SUPPORTDB                              UPS_SUPPORT
-    UPS                                 ZOOM                                          ZOOM-SUPPORT
-  ABOUT-ROOT

## More Stuff:

- Marc Paterno (C++ Guru)  bringing Run Control Parameter D0 Code to CDF.
- Developments and Infrastructure for Run II starting to form  basis for needs for other Experiments.

- Finally

- Next reports of  the actual work...