# Presentation: 138330

## How do we get the computing and software we need to do physics analysis?

- D**** Oklahoma Workshop,  12th July 2002.
- Terry Wyatt.
-  (Freed by the generous support of
-  D****-CD to spend 2002 playing truant
-  from Manchester University.)

## How do we get the computing and software we need to do physics analysis?

- And how do we make the computing and software
- (and the data)
- accessible to the maximum number of D**** physicists
- with the minimum number of unnecessary barriers?
- building detector    taking data
-        writing software     analysing data
- We should re-examine our approach to computing and software in this light.

## At this stage of the experiment “doing physics” means:

- calibrate and align detector components
- find problems with detector/trigger/readout
- measure and improve the performance and reliability of the reconstruction code +
- + reconstruction farms
- + data catalogue
- + infrastructure for users to access data
- Measuring cross-sections, doing searches, of interest mainly in so far as they promote the above

## This talk is not:

- A review of recent improvements in our computing and software infrastructure
- See plenary talks by Amber and Suyong on first day of plenary
- See talks given at the General Algorithms meeting (session B8 of this workshop)
- A discussion of longer term computing and software
- run 2b computing/studies
- off-site specific issues – e.g., Regional Analysis Centres
- Focussed on short term (3-month) goal of this workshop: getting D****** **ready to do physics
- An attempt to make concrete proposals of places where extra effort will be cost-effective

- _This talk is:_

## You can’t always get(the computing and software)what you want,but if you try sometimes, you might find, you get (the computing and software)what you need.With apologies to Sir Michael Jagger .....

## ..... and thanks to:

- Michael Begel, Chris Barnes, Amber Boehnlein, Volker Buescher, Yann Coadou, Tom Diehl, Mike Diesberg, Hal Evans, Frank Filthaut, Gaston Gutierrez, John Hays, Mike Hildreth, John Hobbs, Lee Leuking, Harry Melanson, Wyatt Merritt, Michiel Sanders,
- and especially to Marco Verzocchi.
- + some very useful meetings held at this workshop!

## Reco Output in p11, p12 

- DST (150 kB/event – mainly on tape):
- “physics objects”: tracks/clusters + leptons/jets/missing ET
- “calibrated” hit data to allow re-reco
- TMB (thumbnail – 10 kB/event – all events on disk):
- subset of information from subset of physics objects
- packed to save space
- unpacked in “framework” analysis job to re-produce physics objects (lite)
- TMBTree (RootTree – 20 kB/event – produced by groups/users):
- same information as TMB
- stored in root format

- User feedback on new formats needed!
- serious current deficiency – absence of L1/L2/L3 trigger objects in TMB – must be fixed for p13!
- Between now and the October shutdown we can afford to store on disk “TMB+”:
- standard TMB chunk
- still to be determined sub-set of calibrated hit data to allow limited re-reco
- (e.g., allow re-clustering in smt, cft, calorimeter)

## TMB and TMBTree are complementary 

- TMBTree files ~ twice as large per event because it contains “derived” quantities
- which have to be re-calculated when reading TMB file
- TBMTree allows convenience of interactive root and interpreted C++
- but in practice serious analysis requires:
- batch processing using compiled code
- “certified” analysis tools need to be:
-  independent of input format (e.g.,  jet_corr)
-  propagated back into reco!
- TMB allows flexibility and power of D**** framework!

## d0tools provides an easy way to run simple jobs, but ....
Pressure to keep up with weekly releases impossible unless you’re a full-time code developer
Documentation:
D run II coding for professors
Framework Users Guide
Event Data Model (EDM)
Run Control Parameters (RCP)
CVS
.... but much less knowledge than this is needed to run a simple analysis job 

- Pressure to keep up with weekly releases impossible unless you’re a full-time code developer
- Documentation:
- D****** **run II coding for professors
- Framework Users Guide
- Event Data Model (EDM)
- Run Control Parameters (RCP)
- CVS
- .... but much less knowledge than this is needed to run a simple analysis job

- _High threshold to get started with D__****__ framework_

- Level appropriate
- for designers of
- new packages ....

## What we need is: 

- d0tools utility to run DST/TMB analysis job
- skeleton user analysis framework package:
- loops over each physics object type
- produces example histogram/roottuple/event statistics output
- Essential documentation:
- “D**** analysis computing primer”
- containing minimum necessary information on framework, EDM, RCP, CVS, SAM, etc, to do analysis
- documentation of access methods for physics objects (!)
- detailed documentation on physics object contents:
- how are the quantities calculated?
- which corrections are applied?
-     and we need it very soon!

## Other Infrastructure Issues

- Performance, quality monitoring
- SAM
- Streaming
- Trigger simulations
- Monte Carlo simulation – tuning to data
- Event display

## Peformance monitoring

- What we want is:
- Code releases that are thoroughly tested (don’t crash and produce correct results)
- Reliable cataloguing and delivery of data
-    (both to production farms and to users)
- 100% efficiency for reco farm to process recorded data and produce output files containing correct results
- Actually, this is what we need!

## How do we work towards achieving these goals?

- Harry’s two monitoring projects:
- histogram and statistics-based
- (large overlap with trigger/physics examines)
- error message based
- (need to eliminate the huge number of spurious error messages currently produced by reco)
- Start with systematic testing of new code releases using standard set of test samples.
- Progress to systematic monitoring of SAM and reco farm performance (quantity and quality)
- significant role for offline and/or data-quality shifters
- PLEASE volunteer!
- a great way for newcomers to learn about the detector and the data
- excellent preparation for doing a physics analysis

## Other SAM Issues

- “My files in the SAM cache got deleted before my job ran”
- PLEASE use sam submit or d0tools
- It really is very easy!
- SAM will try to keep frequently used datasets in the cache
- but can do so only if it knows that you are using them!
- PLEASE can we have SAM running on clued0 soon!

## SAM Issues (continued)

- Good, bad and ugly data files and SAM datasets:
- we need to be able to classify, e.g., as “current”, “obsolete”, “junk”
- In the short term it would be INCREDIBLY USEFUL if one or two knowledgeable people would put together a web page summarizing the current best available MC samples and their SAM dataset names!
- the return on this investment would be huge

## Streaming

- L3 filter code in p12 is ready for streaming
- Soon to be available:
- two alternative streaming schemes
- tools to allow physics groups to test performance of streaming
- Joint meeting of analysis tools, ORB, trigger board (and anyone else who is interested) to review results

## Trigger Simulations

- Will become increasingly important as luminosity increases and trigger/DAQ system is fully commissioned
- for general physics users – not just trigger nerds!
- Trigsim has been rather fragile
- Needs to become part of standard MC farm production in p12
- use simple generic trigger list to create all (sufficiently well tested) L1/L2/L3 objects for every event
- Documentation of L1/L2/L3 objects is poor or non-existent in many cases
- Increase pool of people able to use trigger database to define new trigger lists

## Monte Carlo simulation – tuning to data

- More realistic simulation of calorimeter on the way (finally)
- QCD and Zleptons urgently needed
- Need simple fixes to other problems
- e.g., permanently dead SMT ladders
- The Monte Carlo we generate in the next six months is disposable
- we should try to simulate as realistically as possible the detector and data quality we have now

## Event displays

- D****VE and D****Scan
- less influence on debugging detector and reco than I would have expected
- need tutorials at FNAL every month or so
- Need displays of:
- fired triggers
- trigger physics objects
- a good place to make use of the proposed “common interface” to L1/L2/L3/reco/MC objects?
- Non-linear scale in r
- Option to chose objects on input vs. re-reco
- User needs choice of colour options:
- by energy, momentum
- to indicate jets, etc

## Action Items

- _Now_
- p12 development is finished
- exceptions must be specifically approved by Harry Melanson
- _End July_
- p12  L3 and reco farms (no excuses!)
- p12  MC farms subject to possible delay of one month, because one person goes on holiday!

## Action Items (continued)

- _Two weeks’ time_
- d0tools utility to run new example framework analysis package on TMB, DST
- first version of “D**** analysis computing primer”
- documentation of access methods for reco physics objects
- web-page with current best MC datasets
- _Beginning Oct_
- detailed documentation of reco physics object contents
- p13 build starts
- collaboration week: day of tutorials on framework and TMBTree analysis, SAM, etc?

## Action Items (continued)

- _End Oct_
- p13  L3, reco and MC farms
- if shut down delayed we might be able to get significant re-processing done
- reco farm upgrade up and running

- As we move towards becoming a physics experiment:
- Computing and software are no longer primarily a question for (computing and software) professionals
- They are an integral part of our physics analysis activities ...
- ... and need closer link with our physics coordination
- ... and I think we need two physics coordinators to carry out this extended role