# Presentation: 922928

##  Global Monitoring 

- Pushpa Bhat
- Fermilab
- Shifter Tutorial      Feb. 3, 2003

## Goals: 
Monitor detector, trigger and physics information in real time, in a random sampling of data, to look for any detector/trigger problems
Monitor trigger rates
Spot problems if any and investigate causes
Provide baseline data quality evaluation
Global Examines
Physics, Trigger 
Trigger Rates 
LmTrigger
There are other examines run by detector shifters:
Detector Examines: CAL, SMT, MUO, CFT, CPS, FPS
Trigger Examines: L1CAL, L1CTT
http://www-d0.fnal.gov/~pushpa/GM/							

- Monitor detector, trigger and physics information in real time, in a random sampling of data, to look for any detector/trigger problems
- Monitor trigger rates
- Spot problems if any and investigate causes
- Provide baseline data quality evaluation
- Global Examines
- Physics, Trigger
- Trigger Rates
- LmTrigger
- There are other examines run by detector shifters:
- Detector Examines: CAL, SMT, MUO, CFT, CPS, FPS
- Trigger Examines: L1CAL, L1CTT
- http://www-d0.fnal.gov/~pushpa/GM/

**Global Monitoring**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Detector Examines     (not part of GM)

- Basic raw data information
- Muon
- Hit maps, Times, Address errors
- Calorimeter
- EM, Hadronic energies, ET distributions, occupancies
- Vs Eta,phi
- SMT
- Barrel, fdisk, hdisk cluster charges
- CFT
- Maps of fired fibres, occupancies in layers, hit multiplicities in layers & sectors

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## MuoExamine

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Global Examines

- P. Bhat

**Global Examines**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Global Monitoring Tools

- Physics Examine
- Reco+RecoAnalyze+macros
- M. Sanders, (PB)
- Trigger Examine
- D0Analyze +macros
- E. Cheu, J. Steele, M. Sanders
- GMGui
-  Python GUI that integrates all GM tasks
- J. Webb, PB
- GMbrowser
- Customized root browser
- H. Prosper, PB
- LmTrigger Gui
- Trigger rates
- Han Do, (J. Park)

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GM Tools (contd.)

- Web Monitoring of PhysEx and TrigEx histograms
- DØ at work  RunIIa operations & Commissioning  DØ Live: Physics Examine and Trigger Examine
- http://www-d0.fnal.gov/~pushpa/GM/
- Used for remote shifts
- J. Webb, PB
- Run Info, Certification & Checklists
- J. Webb, PB
- GM  Messenger/Communicator
- GM Chat tool + web message board + archival
- J. Webb, PB

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GM Home Page

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GMgui

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GMGui - RunInfo

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Gmgui - Run Info

**Gmgui - Run Info**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GMgui – Physics Examine

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GMgui – Physics Examine (Upgrade)

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Physics Examine Plots

- Primary Vertex Info
- Gtracks
- CFT tracks
- Jets
- Selected jets
- Electrons
- Muons
- J/Psi
- More to come:
- W, Z

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Trigger Examine Plots

- L1 Muon triggers
- L2 Global jets
- L2 CAL jets
- L2 electrons
- L2 muons
- L3 tracks
- L3 jets
- L3 electrons
- L3 taus
- L3 EM_HI_xx
-  ...

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GMBrowser

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## PhysEx: CFT Tracks

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## PhysEx Global Tracks

**PhysEx ****Global Tracks**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## PhysEx: Sel. Jets

**PhysEx: Sel. Jets**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Click on a title to show page Turn Cycling on   

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## PhysEx: J/Psi

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## TrigEx: L2 CALJets

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## TrigEx: L3 Electrons

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## TrigEx: L3 tracks

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Description of Plots

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## LmTrigger GUI

**LmTrigger GUI**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Run Certification

**Run Certification**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GM Shifter Tasks

- Run Physics & Trigger Examines
- Run LmTrigger (trigger rates GUI)
- Run Browsers (will automatically start)
- Monitor Plots
- If there are problems try to find the cause; communicate with detector shifters, detector experts, shift captain.
- Make entries in the log book and include relevant images
- Complete Run Checklist and Run Certification for each run.
- Print out and look at run summary
- If there are operational problems, contact GM expert on-call.
- Complete shift summary in the log book
- Sent out to the d0_examines email list

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GM Experts On-call

- We are adding GM experts on-call to help with operational problems:
- Pushpa Bhat
- Michiel Sanders
- Joe Steele
- Elliott Cheu
- Jason Webb (Gmgui, GM Messenger)
- Expert Duties:
- Respond to page and help with shift operational problems, NOT help resolve detector/trigger problems seen.  The latter is to be taken care of by the GM shifter and detector/trigger experts.
- Read log book and shift summaries
- Report at the D0 Operations meeting

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GM Remote Shifts

- So far, NIKHEF, Saclay, University of Munich, Aachen, Tata Institute have participated.
- Mainly owl shifts from abroad
- Day or afternoon shifts for them
- Would like to extend it to other places in Europe

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## How do they work?
Remote shifter logs onto one of the fnal machines and then does ssh to d0ol33 (d0gm account)
Sends a message through the GM web messenger tool to the shift Captain; messages appear on
http://www-d0.fnal.gov/~pushpa/GM/
Remote shifter uns the Gmgui locally; Starts all the global examine processes from the GUI which pop up windows on the control room GM console
Needs “xhost +d0ol33”

- Remote shifter logs onto one of the fnal machines and then does ssh to d0ol33 (d0gm account)
- Sends a message through the GM web messenger tool to the shift Captain; messages appear on
- http://www-d0.fnal.gov/~pushpa/GM/
- Remote shifter uns the Gmgui locally; Starts all the global examine processes from the GUI which pop up windows on the control room GM console
- Needs “xhost +d0ol33”

**GM Remote Shifts**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Remote shifter monitors plots of PhysEx and TrigEx on the web
Linked on the GM web page and RunII operations page
Communicates problems to the shift captain using gm web messenger.
Makes text entries in the log book using web entry form
Image entries via web will be possible soon. 

- Linked on the GM web page and RunII operations page
- Communicates problems to the shift captain using gm web messenger.
- Makes text entries in the log book using web entry form
- Image entries via web will be possible soon.

**GM Remote Shifts (contd.)**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GM Messenger

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GM Web Message Board (1)

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GM Web Message Board (2)

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## GM Web Message Board (3)

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## How can we improve?

- Enhancements to the GUI
- Run start/end alerts
- Already in place in GMgui
- Run multiple instances of PhysEx, some on specific triggers
- Better training of shifters
- More automation
- Experienced shifters

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Make the monitoring info available to experts in real time:
Root files or plots on the web?
Plots already available in real time
Use a pool of detector/ID experts to provide supplemental monitoring
They can be anywhere in the world
More detailed monitoring with some latency
Possibly run examines/reco on a slightly larger sample; run a suite of macros to diagnose specific problems	
Online-Offline Feed-back Loop:
What problems have been uncovered in data in offline analyses?  How can the knowledge be incorporated into online monitoring?

- Root files or plots on the web?
- Plots already available in real time
- Use a pool of detector/ID experts to provide supplemental monitoring
- They can be anywhere in the world
- More detailed monitoring with some latency
- Possibly run examines/reco on a slightly larger sample; run a suite of macros to diagnose specific problems
- Online-Offline Feed-back Loop:
- What problems have been uncovered in data in offline analyses?  How can the knowledge be incorporated into online monitoring?

**How can we improve****Data Quality Monitoring?**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Detector/trigger/ID/physics groups to designate someone 
To think about what problems might arise and how they might manifest in the data and how we may find them.
To look at what exists and send me info as to what  else is useful to monitor (and send necessary code wherever possible)
Developing a knowledge-based GMExpert system 
An expert writes down what they know and make that knowledge available electronically;  updates as needed
A database of problems, explanations, actions to be taken,  or just a well organized web page, for now!
Alarms

- To think about what problems might arise and how they might manifest in the data and how we may find them.
- To look at what exists and send me info as to what  else is useful to monitor (and send necessary code wherever possible)
- Developing a knowledge-based GMExpert system
- An expert writes down what they know and make that knowledge available electronically;  updates as needed
- A database of problems, explanations, actions to be taken,  or just a well organized web page, for now!
- Alarms

**Future Plans**

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## Summary

- Global  Monitoring has proven to be useful in the control room
- Cal hot cells, muon hot cells, CFT problems, trigger problems etc. are uncovered but we can do better
- Basic tools exist
- We need more shifters, particularly physicists experienced in data analysis
- Expert help in developing a knowledge-based Expert System  is needed.
- inputs and participation from Detector, ID and Physics groups
- Feedback loop from Offline Data Quality group to improve online monitoring

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003

## SMT Examine

- P. Bhat                      Global  Monitoring

- Feb. 3, 2003