D0 experiment

[Note]{.underline}: *You are invited to make comments and suggestions by
writing in this paper version. However, if you do that, please sign &
date any comments you write in here, so we can get clarification if
needed. Thanks.*

Calorimeter Shifters' Guide

Table of Contents

[Introduction 4](#__RefHeading___Toc59549866)

[How to use this Guide 4](#__RefHeading___Toc59549867)

[Shifter Tasks 6](#__RefHeading___Toc59549868)

[Contacting the on-call expert 6](#__RefHeading___Toc59549869)

[Starting you shift 6](#__RefHeading___Toc59549870)

[Before a store 8](#__RefHeading___Toc59549871)

[At the Start of a Run 8](#__RefHeading___Toc59549872)

[During a Run 9](#__RefHeading___Toc59549873)

[At the End of a Run 10](#__RefHeading___Toc59549874)

[Taking a Pedestal Calibration Run
10](#taking-a-pedestal-calibration-run)

[Taking a pulser pattern ramp run 15](#__RefHeading___Toc59549876)

[If a BLS power supply trips 16](#__RefHeading___Toc59549877)

[Locations of Documentation 18](#__RefHeading___Toc59549878)

[Glossary 19](#glossary)

[How to? 21](#__RefHeading___Toc59549880)

[Getting Help 21](#getting-help)

[Documentation 22](#documentation)

[Logging on 22](#logging-on)

[Window matters 23](#window-matters)

[Printing 24](#printing)

[Useful Tools 24](#useful-tools)

[Taker 26](#taker)

[Electronic Logbook 26](#electronic-logbook)

[GUI's: generic how to 28](#guis-generic-how-to)

[Supply GUI's 29](#supply-guis)

[How to grab the file before SAM gets it
30](#how-to-grab-the-file-before-sam-gets-it)

[Pulser GUI's 30](#pulser-guis)

[Control Software 31](#control-software)

[Monitoring 32](#monitoring)

[Expert Pages 35](#shifter_task_chapter)

[Calorimeter Experts 35](#__RefHeading___Toc59549896)

[ICD Experts 35](#__RefHeading___Toc59549897)

[Trigger Experts 35](#__RefHeading___Toc59549898)

[Other Experts 36](#__RefHeading___Toc59549899)

[Check List 36](#check-list)

[Pulser Run 36](#__RefHeading___Toc59549901)

[Delay Ramp Run 37](#__RefHeading___Toc59549902)

[DAC Ramp Run 39](#__RefHeading___Toc59549903)

[DAC + Pattern Ramp Run 40](#__RefHeading___Toc59549904)

[Double Digitization Run 42](#__RefHeading___Toc59549905)

[Triple Digitization Run 43](#__RefHeading___Toc59549906)

[ICD LED Pulser Run 43](#__RefHeading___Toc59549907)

[BLS trip 47](#__RefHeading___Toc59549908)

[Troubleshooting Guide 49](#__RefHeading___Toc59549909)

[Software problems and failures 49](#__RefHeading___Toc59549910)

[Calorimeter Power Supply Monitor Display GUI
49](#__RefHeading___Toc59549911)

[Histoscope 51](#__RefHeading___Toc59549912)

[Data Taking Problems 52](#__RefHeading___Toc59549913)

[Calib Problems 53](#__RefHeading___Toc59549914)

[Alarm Server/Significant Event Server (SES)
54](#__RefHeading___Toc59549915)

[Downloading & Creating Pedestals 54](#__RefHeading___Toc59549916)

[Channel Archiver 56](#__RefHeading___Toc59549917)

[Hardware Problems and Failures 56](#__RefHeading___Toc59549918)

[Preamp Low Voltage Power Supplies 56](#__RefHeading___Toc59549919)

[High Voltage 57](#__RefHeading___Toc59549920)

[Calorimeter Workstation 57](#__RefHeading___Toc59549921)

[Power Outage 59](#__RefHeading___Toc59549922)

[Overview 59](#overview)

[Preparing for a power outage 59](#preparing-for-a-power-outage)

[Restarting Computers & GUIs 60](#restarting-computers-guis)

[Recovering from a Power Outage 61](#recovering-from-a-power-outage)

[Restarting Power Supplies 62](#restarting-power-supplies)

[If the ADC crates have been Power Cycled
65](#if-the-adc-crates-have-been-power-cycled)

[If the T&C Crates have been Power Cycled
65](#if-the-tc-crates-have-been-power-cycled)

[Detector Performance Monitoring Software
67](#__RefHeading___Toc59549930)

[Overview 67](#overview-1)

[Cal_Elec 67](#cal_elec)

[Overview 67](#overview-2)

[Running the software for debugging
67](#running-the-software-for-debugging)

[Running the software to monitor a ramp
68](#running-the-software-to-monitor-a-ramp)

[Running the software for monitoring
69](#running-the-software-for-monitoring)

[Error messages 69](#error-messages)

[Cal_Examine 69](#cal_examine)

[Overview 69](#overview-3)

[Running the Root version of the software
70](#running-the-root-version-of-the-software)

[Running the Histoscope version of the software
70](#running-the-histoscope-version-of-the-software)

[Histoscope Error messages 71](#histoscope-error-messages)

[Monitoring and Control Software 72](#__RefHeading___Toc59549943)

[Overview 72](#overview-4)

[Calorimeter Global Monitor 72](#calorimeter-global-monitor)

[Overview 72](#overview-5)

[GUI Features 72](#gui-features)

[Crate Monitoring 72](#crate-monitoring)

[Overview 72](#overview-6)

[GUI Features 72](#gui-features-1)

[Calorimeter Pulser 72](#calorimeter-pulser)

[Overview 72](#overview-7)

[Starting the Pulser GUI 73](#starting-the-pulser-gui)

[Downloading a pulser 73](#downloading-a-pulser)

[Checking that the download was successful
74](#checking-that-the-download-was-successful)

[How to make Global settings 75](#how-to-make-global-settings)

[How to set a DAC/Delay Ramp 75](#how-to-set-a-dacdelay-ramp)

[How to create a picklefile 76](#how-to-create-a-picklefile)

[How to choose a predefined pattern
76](#how-to-choose-a-predefined-pattern)

[Some other things you can do 76](#some-other-things-you-can-do)

[GUI Features 79](#gui-features-2)

[HV Control 79](#hv-control)

[Overview 79](#overview-8)

[GUI Features 79](#gui-features-3)

[Hot Cell Killer 79](#hot-cell-killer)

[Overview 79](#overview-9)

[Channel Hierarchy 79](#channel-hierarchy)

[Data Flow 81](#data-flow)

[Step-by-Step Hot Cell Killing 81](#step-by-step-hot-cell-killing)

[Channel Archiver 83](#channel-archiver-1)

[Overview 83](#overview-10)

[Setup 83](#setup)

[Starting a new Archive 84](#starting-a-new-archive)

[Stopping the Archive 84](#stopping-the-archive)

[Remove previous data 84](#remove-previous-data)

[Move Current data to previous 84](#move-current-data-to-previous)

[Determining if the Archiver is Running
84](#determining-if-the-archiver-is-running)

[Restarting a Stopped Archive 85](#restarting-a-stopped-archive)

[Making a CDROM copy of the data 85](#making-a-cdrom-copy-of-the-data)

[Viewing the Archive 85](#viewing-the-archive)

[Downloading 87](#__RefHeading___Toc59549981)

[Calorimeter System Overview 88](#__RefHeading___Toc59549982)

[Preamplifier 88](#preamplifier)

[BLS 88](#bls)

[Pulser 88](#pulser)

[ADC 88](#adc)

[Timing and Control 88](#timing-and-control)

[LaR Monitoring 88](#lar-monitoring)

[Preamp 89](#__RefHeading___Toc59549989)

[Overview 89](#overview-11)

[Design 89](#design)

[Performance 89](#performance)

[Details 89](#details)

[Safety 89](#safety)

[BLS 90](#__RefHeading___Toc59549995)

[Overview 90](#overview-12)

[Design 90](#design-1)

[Performance 90](#performance-1)

[Details 90](#details-1)

[Safety 90](#safety-1)

[Pulser 91](#__RefHeading___Toc59550001)

[Overview 91](#overview-13)

[Design 91](#design-2)

[Performance 91](#performance-2)

[Details 91](#details-2)

[Safety 91](#safety-2)

[ADC 92](#__RefHeading___Toc59550007)

[Overview 92](#overview-14)

[Design 92](#design-3)

[Performance 92](#performance-3)

[Details 92](#details-3)

[Safety 92](#safety-3)

[Trigger 93](#__RefHeading___Toc59550013)

[Overview 93](#overview-15)

[Design 93](#design-4)

[Performance 93](#performance-4)

[Details 93](#details-4)

[Safety 93](#safety-4)

[Timing and Control 94](#__RefHeading___Toc59550019)

[Overview 94](#overview-16)

[Design 94](#design-5)

[Performance 94](#performance-5)

[Details 94](#details-5)

[Safety 94](#safety-5)

[Rack Monitoring 95](#__RefHeading___Toc59550025)

[Overview 95](#overview-17)

[Design 95](#design-6)

[Performance 95](#performance-6)

[Details 95](#details-6)

[Safety 95](#safety-6)

[High Voltage 96](#__RefHeading___Toc59550031)

[Overview 96](#overview-18)

[Starting the GUI Software 96](#starting-the-gui-software)

[Starting the Cal/ICD HV Monitoring GUI
96](#starting-the-calicd-hv-monitoring-gui)

[Starting the Cal/ICD HV control GUI
96](#starting-the-calicd-hv-control-gui)

[Resetting the HV (ICD and Calorimeter)
97](#resetting-the-hv-icd-and-calorimeter)

[Calorimeter High Voltage 98](#calorimeter-high-voltage)

[ICD High Voltage 98](#icd-high-voltage)

[Details 99](#details-7)

[LAr Monitoring 105](#__RefHeading___Toc59550040)

[Overview 105](#overview-19)

[Design 105](#design-7)

[Performance 105](#performance-7)

[Details 105](#details-8)

[Safety 105](#safety-7)

[Appendix A -- Rack Maps 106](#__RefHeading___Toc59550046)

[On the Calorimeter 106](#on-the-calorimeter)

[Preamps 106](#preamps)

[Preamp Low Voltage Power Supplies
106](#preamp-low-voltage-power-supplies-1)

[HV distribution 106](#hv-distribution)

[Under the Detector 106](#under-the-detector)

[BLS 106](#bls-1)

[BLS Low Voltage Power Supplies 106](#bls-low-voltage-power-supplies)

[In the Moveable Counting House (MCH)
106](#in-the-moveable-counting-house-mch)

[ADC 106](#adc-1)

[ADC Low Voltage Power Supplies 106](#adc-low-voltage-power-supplies)

[Timing and Control 106](#timing-and-control-1)

[CETEC Low Voltage Power Supply 106](#cetec-low-voltage-power-supply)

[HV Power Supplies 106](#hv-power-supplies)

[L1 Cal Trigger 106](#l1-cal-trigger)

[In the Control Room 106](#in-the-control-room)

[Appendix B -- Safety Documentation 107](#__RefHeading___Toc59550062)

[Appendix C -- Scripts Explained (for Experts)
108](#__RefHeading___Toc59550063)

[Overview 108](#overview-20)

[The Scripts 108](#the-scripts)

[Downloading Pedestals 108](#downloading-pedestals)

[Taking pedestal Calibration Runs
109](#taking-pedestal-calibration-runs)

[Checking the memory on the T&C board after it is power cycled
109](#checking-the-memory-on-the-tc-board-after-it-is-power-cycled)

[Make sure we are running in the correct mode for data taking
109](#make-sure-we-are-running-in-the-correct-mode-for-data-taking)

[Set the Timing and Control system to fixed cell mode, 0x889
110](#set-the-timing-and-control-system-to-fixed-cell-mode-0x889)

[Starting the HV GUI's 110](#starting-the-hv-guis)

[Run the HISTO version of Calorimeter Examine
110](#run-the-histo-version-of-calorimeter-examine)

[If the FPGA code needs to be downloaded into the T&C crate
113](#if-the-fpga-code-needs-to-be-downloaded-into-the-tc-crate)

[Appendix D - Calorimeter Checklist 116](#__RefHeading___Toc59550074)

[Index 116](#__RefHeading___Toc59550075)

[]{#__RefHeading___Toc59549866 .anchor}Introduction

this User's Guide is intended to serve the needs of shifters during
their D0 Calorimeter shift, and to provide documentation for the experts
to help debug and repair problems with the Calorimeter system. The term
"Calorimeter shifter" now has been expanded to include not only the
calorimeter, but also the ICD. Do not panic just because this is a huge
document -- as a shifter you should need only to use the first 40 pages
or so (ok -- that is a lot, but there is a lot to say). The other
material is mostly for expert use or if you are interested in more of
the details.

This guide is a compilation of notes, e-mails, D0 notes, web pages, etc.
It should provide all the information you need to run a Calorimeter
shift. It is organized in such a way as to start with essential
information such as contacts (page [14](#_Ref517805354)), an FAQ (page
[20](#_Ref9341513)) a quick guide to running and taking calibrations
(page [Error: Reference source not found](#_Ref9341491)), followed by a
description of the essential shifter tasks when there is beam (page
[Error: Reference source not found](#_Ref9341583)), starting with the
very basics of logging on, and the standard monitoring tasks to be run
by shifters, and when there isn't beam (page [Error: Reference source
not found](#_Ref9341632)). To help you resolve problems, we have
included a Troubleshooting guide to common problems (page
[14](#_Ref517805354)) -- this will likely grow as the run progresses. As
a reminder of the available troubleshooting tips, we have placed
suggested troubleshooting page numbers in the left margins throughout
this manual where appropriate. Later chapters give the details for a
variety of software programs and hardware. Some of the more arcane
details on subjects such as error codes are left for the Appendices. As
a shifter you should normally have to read only the Shifter Task chapter
starting [73](#shifter_task_chapter).

Throughout the document you should be able to find links to other
relevant sections or web links.

This document is created in MS Word 2002 (in Office XP), and is
converted to pdf format (preserving all links using PDFMaker) using
Adobe Acrobat 6. The master word file is kept in
<http://www-d0online.fnal.gov/www/groups/cal/Manual.pdf>. Please send
corrections and additions to [mailto:tuts@fnal.gov?subject=Cal Shifter
manual](mailto:tuts@fnal.gov?subject=Cal Shifter manual), or
[mailto:parua@fnal.gov?subject=Cal Shifter
manual](mailto:parua@fnal.gov?subject=Cal Shifter manual)

# How to use this Guide

This guide is designed to help you on your shift. Here is how you should
use it. First locate the section that is relevant to your shift, e.g.
Getting Started, or When there is Beam, etc. Within a section, the
numbered steps should serve two purposes: (a) the boldface lead sentence
should be a summary of the step (so if you are an experienced shifter,
that should be all the info you need, and there is no need to read the
attached paragraph!); (b) if you are inexperienced or have forgotten
what the step means, then go ahead and read the paragraph, which will
give you the detailed instructions to follow. **Do NOT read it like a
novel every time** -- use it as a guide to remind you of the necessary
steps.

Once you are more expert you can use the Quick Guide in the previous
chapter.

Also note that in some cases you will be referred to pages outside of
this chapter for more details. You will need the full manual to see
those pages. The margin notes should point you to sections in the
troubleshooting guide that may help.

  ---------------- ---------------- ------------ ------------ ------------
                                                              

                                                              

                                                              

                                                              

                                                              

                                                              

                                                              

                                                              

                                                              

                                                              

                                                              

                                                              
  ---------------- ---------------- ------------ ------------ ------------

# 

# 

  --------------------------------------------------------- -------------
                                                            

                                                            

                                                            

                                                            

                                                            

                                                            

                                                            

                                                            

                                                            

                                                            
  --------------------------------------------------------- -------------

#  

# 

# 

  ----------------------------------- -----------------------------------
                                      

                                      

                                      

                                      

                                      

                                      
  ----------------------------------- -----------------------------------

[]{#__RefHeading___Toc59549868 .anchor}Shifter Tasks

I Contacting the on-call expert

In case of problem, contact first the on-call expert. He will know how
to solve many problems and who else to call. If you don't find him,
contact people below in the order of the list. If you cannot find any of
the people listed here, try people listed in the Expert Task Lists.

  ---------------- ---------------- ------------ ------------ ------------
  Name             Office phone     Home phone   Pager        Cell phone

  **On call                                      **(630)      
  expert**                                       218-4777**   

  **Robert         \(630\) 840-2694 \(312\)      **(630)      **(630)
  Zitoun**                          341-1696     266-0634**   399-0024**

  Pierre Pétroff   \(630\) 840-6447 \(630\)                   
                                    840-3571                  

  Nirmalya Parua   \(630\) 840-6792 \(630\)      \(630\)      
                                    985-1160     218-8489     
  ---------------- ---------------- ------------ ------------ ------------

# Starting you shift

First you should login in the logbook and check if everything is
correctly setup. If something is not, have a look in the next section.

1.  Locate the monitors and orient yourself. There are three principal
    calorimeter monitors which by convention we call ***Monitor #0***
    (bottom center), ***Monitor #1*** (bottom left), ***Monitor #2***
    (top left). Note that each monitor has a labeled workspace in the
    lower right hand corner, which we will refer to in the following
    sections.

2.  Log into the electronic logbook. The logbook is normally open on
    Monitor #0 so you can record your activities during the shift. You
    should ***log*** yourself ***in*** (this normally logs out the old
    shifter) with your username and password so that the entries will be
    tagged with your name.

3.  Locate the calorimeter monitoring GUI's. Normally the standard GUI's
    should be running on Monitor #1 (***RMI*** , ***IOC***, ***Alarm***
    and ***Global Monitor***) and #2 (***Supply***, ***Crate Monitor***
    (also called ***T&C*** or ***tandc***), ***hv,*** and ***Pulser***).
    The color of the entry tells you the state: green is what it should
    be; yellow means that it is slightly out of tolerance (but
    acceptable); pink means that it is definitely out of tolerance (note
    that there may be pink items that do not trigger a MAJOR alarm).
    Most important are

    a.  the ***Alarm*** GUI (Monitor #2). Each line in this GUI
        corresponds to a page in the ***Supply*** GUI or to the ***hv***
        GUI.

    b.  the Calorimeter Power Supply Monitor Display or ***Supply*** GUI
        (Monitor #1). On this GUI, you will see tabs for the various
        power supplies: Preamp, BLS N (BLS North), BLS C (BLS Central),
        BLS S (BLS South), BLS BCK N (BLS backplane North), BLS BCK S
        (BLS backplane South), ADC Temp (temperatures in the ADC power
        supplies), PA Temp (preamp temperatures), Fanout, ADC, Pulser,
        PLS Mode (pulser mode), Mode, Mode Shift.

    c.  The ***hv*** GUI

4.  Make sure that all low voltage and high voltage power supplies are
    on and working. You should continuously check the status of the
    calorimeter. Look at the Alarm GUI on Monitor #2 and check to see
    that there are no persistent (i.e. is there for more than 30
    seconds) ***MAJOR*** alarms (which appear in pink). If there are
    persistent ***MAJOR*** alarms, then use the relevant GUI/page to
    better locate the problem. You can check against the latest
    reference screen shots at
    <http://www-d0online.fnal.gov/www/groups/cal/Monitor/test.html>.
    Make an entry in the logbook and reset the power supply which
    tripped (for HV, see the details 193). If the reset fails, report to
    the on call expert who might tell you what to do, such as to
    acknowledge or disable the alarm. If you want more details on the
    status of any "box" in the display (for example you want to find out
    specifically what made a box turn red), then left-click on the box
    and it will pop up a detailed list with the offending item in red,
    that will help you figure out where to look for the problem.

5.  Monitor preamplifier temperatures. Go to the ***Supply GUI*** in
    Monitor #1. Click on the ***PA Temp*** tab. Capture this page as
    ***/home/d0cal/temp.gif*** (see How To page xxx), print it
    ***printbw*** ***/home/d0cal/temp.gif***. Get the small white binder
    on the left of Monitor #2. Check the columns against the reference
    temperatures (on the cover of the binder). Report in the logbook and
    contact an expert if you see any change more than 2 degrees.

6.  Make sure dq_calo and dq_monitor are running.

    a.  From any xterm window on ***d0ol45***, type ***ps -C
        dq_calo_x***. You should see at least one line with dq_calo_x.
        If not, type ***start_cal dq_calo*** in any xterm window, this
        will start it.

    b.  The dq_monitor GUI should be running on Monitor #0. If not, type
        ***start_cal dq_monitor*** in any xterm window in Monitor #0. In
        the taskbar, you will see the current run number and quality.
        There are 3 frames in that window: the list of available histos,
        the ROOT histograms and the data quality info (text).

7.  Locate the taker. The taker should be running in a window of Monitor
    #0. If you don't see any, type ***ps --u d0cal*\|*grep*** taker. If
    you see taker.x, then look better, the taker is somewhere. If you do
    not see taker.x, start a taker as indicated in the How To section.

# Before a store

# 

> T his step assures that the latest pedestals are downloaded and the
> electronics is set in the correct state. You can of course do this at
> any time, such as when you come on shift and you are unsure of the
> state that the last shifter left the electronics in. To be
> conservative you must carry out this step about an hour
> [before]{.underline} the first run of a store. As this download is a
> severe load on coor for \~1min, get the shift Captain's permission (he
> generally should give you priority for the download) and inform the
> DAQ shifter to [free the calorimeter crates]{.underline} (if they are
> not yet free).

1.  Download "cal_prepare_for_run". Go to the taker window, click on
    ***Modify***, and then click on ***Change Trigger***. Find file
    ***commissioning/cal/cal_prepare_for_run-xxx***. It will tell you
    ***download in progress***. This will among other things download
    the pedestals. If you see a message such as "15 good, 1 bad", go
    again to ***Modify*** and click on ***Revalidate***. If you see a
    red line stating "Download completed ... but with ERRORS" and few
    lines above "Saw disconnection from devdnl", tell the DAQ shifter to
    "restart comics". Then do the Modify/Revalidate procedure described
    above. For any other error, check with the DAQ shifer.

2.  Verify the calorimeter is ready to take data. In the Supply GUI
    (Monitor #1), check the ***Mode Shift tab***: All columns should be
    green except may be the OCC column (TC Mode = 0x10 except the last
    lign T&C CTRL 0x4C = 0x8089, status = 0x10, ADC ERR = 0x0, BLS Mode
    = Normal , ADC Mode = Sel 0 Sign Sup, Pulser Off). If you see
    ***Pulser On***, go to the pulser GUI and press Shutdown. Pulser
    column should turn to off.

3.  Check that the pulsers are really off. You do that by selecting the
    ***PLS mode*** tab in Supply GUI. You should basically see 0
    everywhere. Check especially that the ICD pulser is off by verifying
    that the last row called ***ICD LED*** is set to zero.

4.  Don't free the trigger.

5.  Verify that dq_calo and dq_monitor are running. See step 6 of
    "Starting your shift" section.

# At the Start of a Run

1.  Start L1CalExamine. This program should run on ***d0ol23*** and not
    on d0ol45 which is kept for dq_calo. Look if there is already an
    xterm window of Monitor #0 logged on d0ol23. If not, type
    ***start_cal l1examine1***. This will bring you to d0ol23. Then type
    ***start_cal l1examine2***. It will start killing any histoscope
    version running and open a new histoscope, then start L1CalExamine.
    Once this is running, type ***init*** followed by ***start***. You
    will see it connect to the distributor, initialize and events will
    start flowing. Connect histoscope to the data by selecting the
    ***File/Process*** pull down menu and selecting ***Connect*** ***to
    process***..., that will load in something like ***L1Cal-histos
    Histograms (d0cal)xxxx***. Highlight/select that file and left click
    the ***Connect*** button. Now you should see in the ***Sub
    Categories*** list that there is a single file selected (if there
    are more, you want the one called ***L1CALEXxxx.HBK***). Double
    click on this name and a whole set of histogram names will appear in
    the right-hand pane. Highlight the first 9 and left click on the
    ***View*** ***Multiple*** button.

# During a Run

Your main job will be to see that the calorimeter is running smoothly.
You do that by monitoring its status through a number of programs and
control GUIs.

1.  Keep an eye on the alarms

2.  Monitor the trigger.

    a.  Keep an eye on the histoscope plot.s You can get more
        information on L1 Cal Examine by looking (careful since the
        detailed directions may be obsolete) at
        <http://www-clued0.fnal.gov/~kehoe/l1cal/L1CalExamine.txt>.

    b.  Adjust the L1 Cal Examine thresholds if necessary. You may find
        that the L1 Cal Examine plots are not useful because it looks
        like the thresholds are set incorrectly for these particular
        running conditions. If so, you can change them. in the file
        ***L1CalExamine.rcp***. ***Open an xterm window***, then type
        ***cd /online/examines/development/l1calex/l1cal_examine/rcp***.
        Now edit that file using for example the command ***xemacs
        L1CalExamine.rcp***. Locate the lines with ***EMcut***,
        ***TOTcut***, and ***HDcut*** and change the corresponding
        thresholds accordingly -- unfortunately I don't know what
        accordingly is! (I hear 3.0 looks like a good value, but then
        the noise on the plots changes to 4.0 or 5.0). If it is a
        reasonable time of day and you need help, call the L1 Cal
        Expert. When you are done editing, click on the ***File***
        button and select ***Save L1CalExamine.rcp*** and then ***Exit
        XEmacs***. You will now need to restart L1CalExamine following
        the previous step directions.

3.  

4.  

5.  

6.  

7.  

8.  

9.  

10. 

11. 

12. 

13. 

14. Monitor the data. Keep an eye on the cal_monitor display. Especially
    on

```{=html}
<!-- -->
```
a.  tower occupancy and mean energy 2-D distributions for zero bias and
    jet triggers

b.  METx and METy 1D-distributions for zero bias and jet triggers

c.  the report window.

> For more information, click on the ***Help*** button in the upper
> right corner of the dq_monitor window.
>
> If dq_monitor seems to be stalled, quit and restart it: ***start_cal
> dq_monitor***. If it still seems to be stalled, restart dq_calo. You
> might have first to kill the current dq_calo. So, type ps --C
> dq_calo_x and if you see a dq_calo_x task running, type kill nnnnn,
> where nnnnn is the PID of the first line in the list.

# At the End of a Run

You must enter histograms into the e-logbook

1.  dq_monitor histos. Press ***Print*** in the upper right corner of
    the GUI. A new window appears. Press ***Save***, the file is saved
    as ***/home/d0cal/calex.gif***; you can enter in the logbook. If you
    press ***Print***, you will have a color copy of the screen.

2.  Check for hot cells. In the dq_monitor window, click on ***Report***
    to check for hot cells. Report them in the e-logbook.

3.  L1CalExamine. In the d0ol23 window, type ***import l1cal.gif***.
    Then click on the histogram window. This will put the plots into
    file ***/home/d0cal/l1cal.gif***. Enter it into the e-logbook.

    a.  

# 

# 

# Taking a Pedestal Calibration Run

To take a pedestal calibration run, there should be ***no beam activity
at all***. You will need either about ***50 minutes*** or ***10
minutes*** of no beam time -- in the former case you will take a
***reference*** 10k event run (this is the ***default action***), and in
the latter case a ***short*** 500 event run. Note that the quiet time is
only required for the actual runs, not for creating and downloading the
pedestal file. When you take calibration runs, you will be calibrating
all twelve calorimeter crates at once. You will start by running the
calibration for the BLS ***x8*** gain, then you do it all again for the
***x1*** gain. ***Strategy*** is the following: if you have plenty of
time take 2 reference runs (x8 and x1). If you think you might be short
of time, take a short x8 run then a short x1 run; when they are done,
then start a x8 reference run and next a x1 reference run. If some
reference run cannot be finished, no problem, a short run has been
taken.

1.  Open the Calibration Manager GUI. This must be done as ***d0run***
    ***(not d0cal)*** user. In an xterm window which will be devoted to
    calibration, type:

> *\> start_cal D0run*
>
> *\> setup d0online*
>
> *\> start_calib_manager_gui*
>
> After the calib GUI starts, click on the ***Cal*** tab.
> [Note]{.underline}: NEVER ***exit*** from this GUI if the ***ABORT***
> button is showing in the bottom left of the GUI, as that may leave
> detached processes running which will corrupt the writing to the
> database, and make you think everything looks ok until you check the
> plots. If, by accident, you do exit incorrectly, then start the GUI up
> again and hit the ***ABORT*** button, or wait until you see
> ***CONFIGURE*** (at which point you can click ***Exit***). If that
> fails, call the on-call expert, who may in turn call the Calib expert
> (Taka) to clean up those detached processes.

2.  Get a Taker ready. In the Taker, check that recording is on by
    looking at the upper right hand corner of the taker window, which
    will show ***Recording: on***. If it is set to ***Recording: off***
    then left click ***Modify*** and setting the ***Recording*** box on
    (as indicated by the little red box next to it).

3.  Get the calorimeter and SMT crates.

    a.  You should ask the permission of the Shift Captain to remove the
        Calorimeter crates from the global run, and you should ask the
        DAQ expert to remove them once you have approval.

    b.  For several reasons, we need to take pedestals with the SMT
        being read out. It should not be pulsed to take the data, but
        ***whenever we are not reading it out for more than 1 or 2
        minutes, pulsers should be running (off)***. So, tell the SMT
        shifter that you are going to read their crates out; ask him to
        have the SMT pulsed for now.

4.  Change the trigger. In the Taker, click on ***Modify***, and then
    click on ***Change Trigger***. You will now see the configuration
    window. To make sure you are in the correct directory, click ***UP
    (Left)*** until you are all the way "up" and the directories no
    longer change. Double click on the directory ***calibration***, then
    double click on ***cal***. Now you must choose the appropriate
    trigger:

```{=html}
<!-- -->
```
a.  if you have a long time (\~10 minutes at 20 Hz), double click on
    ***calib_ref_x8*** (10k events);

b.  if you have a short time (\~2 minutes at 20 Hz), double click on
    ***calib_ped_x8*** (500 events);

> Always use the higher version number. It will tell you ***download in
> progress***. After the download is complete (about 10 seconds), make
> sure that all 12 ***Processing Status*** lights are ***dark blue*** in
> the Calib Manager GUI. If they aren't, there is no point starting, so
> you should call the on-call expert.

5.  

6.  Start the run. Before starting the run, ask now the SMT shifter to
    ***stop the SMT*** ***pulsers*** ***(off)*** (remember to have the
    SMT pulser on if data are not being taken). When this is done, click
    on ***Start*** (lower left long button in Taker) to begin the run. A
    pop-up ***Begin Run*** window will appear. Fill it in with your name
    in the Shifter box, and ***ref or peds x8*** (depending on the run
    type) in the Comment box, then click ***OK***. That box disappears
    and the run starts. If you have trouble, check with the DAQ shifter.

7.  Check the calorimeter status. On the Mode Shift page of the Supply
    GUI (Monitor #1), the 4 first columns should be green (Normal, 0x10,
    0xyyyy, 0x0). Next 2 columns should be blue (Force x8 and
    Unsuppressed or Raw). Next 2 columns should be green (0xzzz,
    PulserOff). Don't worry for the last column if it appears pink).
    Check that the last line is green (x8089, 0x10, 0xyyyy).

8.  Watch the Run. When the run starts, the ***Processing Status***
    boxes turn ***from dark blue to light blue***. In the lower line of
    the Taker, you will not see the event number increasing: it will
    stay equal to 0 for the whole run. You must keep checking that the
    run is progressing by

```{=html}
<!-- -->
```
a.  looking at the Mode Shift column L3TRAN: the number should update
    every second or so.

b.  keeping an eye on the ***ADC ERR*** column (the ***4^th^*** one in
    the ***Mode shift*** page) to make sure everything is green.

> You will know that the run is finishing when the ***Processing
> Status*** boxes in the Calib Manager GUI turn ***from light blue to
> green***. Then all 12 ***DB Commit Status*** boxes, starting as grey,
> turn to light blue. A yellow line saying ***Coor is stopping your
> run*** appears next telling that the run is finished. Wait until you
> see the 12 ***DB commit status*** boxes turning to green. The x8 run
> is completed.

9.  If you had a problem in the previous step.

    a.  Ask the SMT shifter to start its pulsers.

    b.  If, before the ***Processing Status*** boxes turn to green, you
        see no more activity in the L3TRAN number, ask first the DAQ
        shifter if something is happening. If you have to restart the
        run, first ***Abort Run*** (lower left button in the Calib
        Manager GUI). That should also stop the Taker (yellow line). Go
        back to step 4.

    c.  If ADC ERR goes to pink, then try clicking the ***Global T&C
        Reset*** and the ***Global ADC Reset*** buttons, if that doesn't
        work, consult with the DAQ shifter, and if that fails restart
        the run (see step 8a).

    d.  If a power supply fails during the run, take a new run starting
        from step 4, but first read page 13.

> If it definitely does not work, call the expert. If you do not see
> anything happening in the calib GUI, the server might need to be
> restarted on d0ol07. Follow the same procedure as in step Error:
> Reference source not found [Error: Reference source not
> found](#calibration_man_server_running). If that does not start it,
> then see if the DAQ shifter can help, or talk to the on-call expert.

10. Record the run number in the logbook. Record the run number (bottom
    line in the Taker) and calibration run type (x8 or x1) in the
    logbook. You will need the information later.

11. Now repeat steps 4 to 9 for x1 gain. Follow steps 4 to 9 above,
    except that you need to change every occurrence of ***x8*** in those
    directions to ***x1***. In particular, that means using the
    ***calib*** ***\_ref_x1*** if you used ***calib*** ***\_ref_x8*** or
    ***calib\_*** ***ped_x1*** if you used ***calib*** ***\_ped_x8***
    trigger file.

12. Start DB Collection. When you have finished taking all these runs,
    ***tell the SMT shifter to run*** pulsers. Then click on ***DB
    Collection*** in the Calibration Manager GUI. The collection is
    finished (this takes about 15min) when the ***DB Collection***
    button pops up again. Pay attention to the run numbers which will
    appear in the calibration xwindow you used in step 2 because you
    will use them in a few moments. NOTE: if you change screens back and
    force, the Calib Manager GUI will not refresh (the window will
    remain gray) until the collection is done.

13. If you had a problem in the previous step (otherwise skip this
    section).\
    \
    ***IF*** an error window pops up\
    \
    ***OR IF*** 12 files with the present date and time are not present
    when you type ***ll
    /online/comics/cal/pic/D0.CAL.\*.ADCC:CONFIG.pic*** in the
    calibration xwindow.\
    \
    ***THEN*** run the following commands from the d0run account: ***cd
    /online/comics/cal*** and then ***NEW_MakePeds.py***. This can also
    take about 5-10min.\
    \
    When you are done, hit ***File/Exit*** in the calib_manager GUI to
    close it. This will stop printing in the d0run window.

14. Free the trigger. This releases the SMT crates as well as the
    calorimeter ones. Soon you will get those back. Tell the SMT shifter
    you are done with his crates.

15. Run Paw. Type the following commands in the xterm window in which
    you did step 2:

> *\> cd /online/comics/cal*
>
> *\> setup cern*
>
> *\>ll peds.txt* (check that the date is the current date, if not call
> the on-call expert)*\
> \> paw\
> *press the *Enter* key to accept the default workstation type (Paw
> window opens)
>
> *exec peds xxxxxx yyyyyyy* and press the *Enter* key (xxxxxx = run
> number for x8, yyyyyy = run number for x1)
>
> press the *Enter* key 12 times to cycle through all plots. Do not look
> at them!
>
> *\>quit*

16. Check the number of bad cells. In the same xterm window as above:

    a.  type ***ll peds_bad.txt***. This file is a link to
        ***/online/comics/cal/ valid/pref_bad_yymmdd-hh:mm:ss.txt*** if
        you took a ***reference run*** (10k evt) or
        ***valid/peds_bad_yymmdd-hh:mm:ss.txt*** if you took a ***short
        run*** (500 evt). Check that the date is the current date, if
        not call the on-call expert.

    b.  type ***less peds_bad.txt***. There is one line per bad
        channels. If the number of channels is greater than ten, you
        should report that information to the cal expert before
        downloading the new pedestals. Columns 1 to 5 are the channel
        address. Next 2 are the recently computed pedestal and noise in
        gain x8 mode. Next 2 are the previous reference values. Next 2
        are the recent pedestal and noise in gain x1 mode. Last 2
        columns are the preamp box and type.

17. Check Paw printout. Print the Paw output by typing ***flpr --q
    dab1_hp8150_d peds.ps***. Get the printout, staple it. Check the
    plots against a set of reference plots which you will find on pink
    paper in the large white binder called ***Calorimeter Reference
    Plots: 02/13/03***. (usually located above the monitors). Note any
    discrepancies in the logbook, in particular

    a.  check the statistics box in the plots to see if the number of
        over/underflows are different from the reference plots), and if
        they look severe enough contact an expert.

    b.  If you see many zeroes in the plots (where there shouldn't be),
        then it is possible that the earlier mentioned potential
        problems of detached Calib Manager processes may have caused
        this -- if so, your only recourse is to contact the on-call
        expert, who in turn may decide to call the Calib expert (Taka).

18. Download cal_prepare_for_run. Unless special runs are planned, do as
    is explained above.

19. Do not free the trigger.

# Taking a pulser pattern ramp run

This type of run will cycle through various pulser patterns, by ramping
through pulser settings. It is assumed that you just downloaded
***cal_prepare_for_run***.

1.  Set the mode. In the Crate Monitoring GUI (T&C GUI),

    a.  click on the purple ***Global T&C Set*** button and select ***Si
        Buff Normal***.

    b.  click on the purple ***Global BLS Mode*** button and select
        ***x8-2ticks***.

    c.  click on the ***Global ADC Mode*** button and select
        ***PedSub***.

2.  In the pulser GUI.

    a.  Select the ***All*** button for the ***Pulser Address***. Note
        you may need to click this twice to actually set it and other
        buttons in this GUI.

    b.  Select trigger bits ***13***.

    c.  Set the DAC value to ***5000***.

    d.  Set the ***Delay All*** box to a value of ***100*** (make sure
        you hit carriage return for this value to be entered).

    e.  Click the ***DAC Ramp or delay*** button -- the ***Request***
        box in the upper right should show a value of **65**.

    f.  Enter the ramp parameters: ***Steps = 1***; ***Step Size = 0***;
        and ***Ev/Steps =*** ***100***.

    g.  Click on the ***Pattern Ramp*** button (near the middle-right of
        the screen), that pops up a new window. In this window, click
        ***Run I Pattern***, then ***Apply*** and finally ***Close***.

    h.  Click ***Download!*** (only once!) to send the ramp request to
        the PIB. If the initial value in the Current Ramp Status pop-up
        window is not zero, then the previous ramp was not stopped. In
        that case stop the ramp in the Ramp Status Window.

3.  Check the settings. In the Power Supply GUI, check the settings on
    the PLS-MODE tab.

4.  Start cal_elec. In an xterm window, type ***start_cal cal_elec***,
    then ***init***, then ***patt*** and finally ***start***. Wait until
    some lines are printed.

5.  Start the run. In the taker,

    a.  select ***Modify Trigger*** and download file
        ***commissioning/cal/cal-all-xxx***.

    b.  set ***Recording on***.

    c.  click on the taker ***Start*** button. You should see the events
        changing in the ramp status window. Note that if the runs gets
        interrupted for some reason and you have to start all over
        again, then make sure to stop the ramp in the ***Ramp Status
        Window***. If you have trouble, check with the DAQ shifter and
        verify that there are no runs either underway or downloaded with
        zero L3-nodes associated. If that is the case, then ask him/her
        to stop the run, free the trigger and restart with a fixed
        (non-zero) number of L3 nodes assigned to their runs.

6.  Watch the changing patterns. As the data is being collected, you can
    open the Calorimeter Power Supply Monitor Display GUI (= Supply GUI)
    to the PLSMode tab where you will see the pulser patterns change.
    Make sure they are changing, otherwise something is wrong.

7.  Stop the Run. After about 3 minutes, when 3200 (100 events times 32
    patterns) events have been collected (as verified at the bottom line
    of the taker or in the ramp status window), you should stop the run
    by clicking ***Stop*** on the Taker. You should also ***Stop*** the
    ramp in the Ramp Status Window. Write the run number in the logbook
    with any other relevant information.

8.  Stop cal_elec. In cal_elec window, type ***pause*** then ***quit***.

# If a BLS power supply trips

1.  Before resetting the supply. Record, in the electronic logbook, the
    following information:

```{=html}
<!-- -->
```
a.  ***Power supply identifier*** -- once you have located the ***pink
    line*** in one of the three tabs in the ***Calorimeter Power Supply
    Monitor GUI*** (marked ***BLS N, BLS C, BLS S***), record the name
    (in the 1^st^ column, something like ***CALx_LVCB_xx_y***).

b.  ***Failure diagnostic*** -- Left click on the cell called ***STAB***
    of the tripped supply (it will be pink). The pop-up list may show
    one (or more) items in pink indicating what failed (e.g. ***B
    overcurrent*** or ***A supply OFF***, etc). Record those items in
    the elog. Click on the same ***pink STAB cell*** to close the pop-up
    box. Repeat the same steps for the ***STCD cell*** and the ***STEF
    cell***.

c.  ***Failure condition*** -- report anything that you think is
    relevant which preceded the trip, e.g. there was a recent change in
    state like the remote was turned on/off, etc.

```{=html}
<!-- -->
```
2.  Reset the supply. Click ***only once*** the Reset button. If the
    fault was ***not overcurrent***, you may hit several times the Reset
    button. If it does not reset, call the expert. Also have a look at
    the Troubleshooting tips on page 87.

3.  

4.  Killing a crate from the readout. If you could not reset the supply,
    you have to take several BLS boards out of a physics run. Advise the
    DAQ shifter and the captain that the relevant crate be
    ***immediately removed from the run***. You want to remove the
    minimal number of channels -- so once you have identified the
    effected channels, you should "kill" those channels. After the bad
    channels are killed, the crate can be put back in the run. Use the
    ***Hot Cell Killer*** to suppress them.

# Locations of Documentation

The calorimeter documentation is scattered across many web areas. We are
in the process of tidying it up and updating it, but for the moment here
are some of the places you should check. Be aware this is a moving
target.

  ------------------------------------------------------------------------------- ------------------
  URL                                                                             Description of
                                                                                  contents

  <http://www-d0online.fnal.gov/www/groups/calmuo/>                               Joint
                                                                                  calorimeter-muon
                                                                                  shifter page

  <http://www-d0online.fnal.gov/www/groups/cal/cal_main.html>                     Cal online

  <http://d0-france.in2p3.fr/WORKING_GROUPS/CALORIMETRY/CALO_CAL/calo_cal.html>   Calibration

  <http://hep.pa.msu.edu/cgi-bin/webcal/webcal.cgi?function=webmonth&cal=CAL>     Shift schedule

  <http://www-d0online.fnal.gov/www/groups/cal/calo_shift_tasks.html>             Today's task list

  <http://d0server1.fnal.gov/projects/calorimeterelectronics/tuts/Manual.doc>     This document on
                                                                                  the web -- the
                                                                                  word file

  <http://d0server1.fnal.gov/projects/calorimeterelectronics/tuts/Manual.pdf>     This document on
                                                                                  the web -- the pdf
                                                                                  file

  <http://d0server1.fnal.gov/projects/calorimeterelectronics/tuts/Manual.htm>     This document on
                                                                                  the web -- the
                                                                                  html file

                                                                                  
  ------------------------------------------------------------------------------- ------------------

#   

# Glossary

Here are some of the terms you may hear while taking a shift. It may
help to translate some of the calorimeter jargon that is used.

  ----------------------------------- -----------------------------------
  Jargon                              Translation

  CRATE MONITORING GUI                Crate Monitoring GUI

  T&C GUI                             Crate Monitoring GUI

  Power Supply GUI                    Calorimeter Power Supply Monitor
                                      Display

  Supply GUI                          Calorimeter Power Supply Monitor
                                      Display

  BLS                                 Baseline subtractor
  ----------------------------------- -----------------------------------

## 

WRONG !!!

  ----------------- ----------------------------------------- -----------
                                                              

                                                              

                                                              

                                                              

                                                              

                                                              

                                                              

                                                              

                                                              
  ----------------- ----------------------------------------- -----------

## Taking a Pedestal Calibration Run

WRONG !!!

  ------------------- ---------------------------------------- -----------
                                                               

                                                               

                                                               

                                                               

                                                               

                                                               

                                                               

                                                               

                                                               
  ------------------- ---------------------------------------- -----------

[]{#__RefHeading___Toc59549880 .anchor}How to?

[]{#_Ref9341513 .anchor}In this section we document a series of
frequently asked questions. If you have one that you think should be
added, let us know. This section has grown so much that we need the
following table to summarize the main areas.

  ------------------------------------------------------------- ---------------------------
  Subject Headings                                              Page

  Getting Help                                                  [55](#getting-help)

  Documentation                                                 [56](#documentation)

  Logging on                                                    [57](#logging-on)

  Useful tools                                                  [58](#useful-tools)

  Electronic logbook                                            [61](#electronic-logbook)

  GUI's and Control Software                                    [65](#control-software)

                                                                
  ------------------------------------------------------------- ---------------------------

# Getting Help

1.  **How do I place a vocal page?**

> *Dial 4074 (or Gosh), wait for the beep. Then place your message twice
> and hang up.*

2.  **How do I page the calorimeter expert?**

> *(a) dial "**9**" to get an outside line, (b) dial **4777** (or 1 630
> 840 4777 if you are not in the 630 area code), (c) wait for the beep,
> (d) enter the number where you want to be called (for example the
> control room would be "8800"), (e) then press "#" or hang up, (f)
> finally wait for the call.* dial "9" to get an outside line followed
> by the number shown below -- note that you may or may not need to use
> the "630", just try it one way or the other.

3.  **What do I do if the expert does not answer?**

> *Try to contact directly Robert Zitoun or any expert in the list.*

4.  **How do I page an expert from on site?**

> *(a) dial "**9**" to get an outside line, (b) dial his pager number,
> (c) wait for the beep, (d) enter the number where you want to be
> called (for example the control room would be "8800"), (e) then press
> "#" or hang up, (f) finally wait for the call.*

5.  **How do I page an expert from off site?**

> *(a) dial his pager number, (b) wait for the beep, (c) enter the
> number where you want to be called, (d) then press "#" or hang up, (e)
> finally wait for the call*

# Documentation

1.  **Where do I find the D0 Calorimeter Online web page?**

<http://www-d0online.fnal.gov/www/groups/cal/cal_main.html>

2.  **Where can I find other documentation (shift schedule, calibration
    info, etc) on the web?**

*See the section on the*
[locations_of_documentation](#locations_of_documentation) *in this
document on page [16](#locations_of_documentation).*

3.  **Where can I find the Cal shift list?**

*You can find it by clicking on "Calorimeter Shift Schedule " on the cal
online web page or at:\
*<http://hep.pa.msu.edu/cgi-in/webcal/webcal.cgi?function=webmonth&cal=CAL>

4.  **I want to make some corrections to this manual. How do I do
    that?**

*Periodically I will check the hardcopy manual that hangs around the
calorimeter shifter's console, so you can write your comments in there
(be sure it is the latest version!). Also, be sure to date and sign your
handwritten comments in case I have questions. If it is urgent, send an
e-mail to*
[mailto:tuts@fnal.gov?subject=Cal%20Shifter%20guide%20comments](mailto:tuts@fnal.gov?subject=Cal Shifter guide comments)
*and cc*
[mailto:parua@fnal.gov?subject=Cal%20Shifter%20guide%20comments](mailto:parua@fnal.gov?subject=Cal Shifter guide comments)*.*

# Logging on

1.  **How do I log on d0ol45 ?**

The username is "***d0cal***" and the password is to be found in the
***D0 Cal User's Guide*** (it is a big white folder) -- it is written on
the front divider labeled Contact Information. Be aware that just
because there is an open window, it may not be a ***d0cal*** window; it
may be a ***d0run*** window. To check to see what account is logged into
that window, just run ***whoami***.

2.  **Where do I find the passwords for the various accounts?**

*Since it is insecure to print passwords in a public document, you can
either ask the DAQ shifter, or the captain, or find the passwords on the
"contact information" yellow front sheet in the D0 Calorimeter User's
Guide (big white binder that usually sits above the monitors of the cal
console).*

3.  **Where do I log on for calorimeter shifts? Where is the cal
    console?**

*The CAL/ICD computer is D0OL45 which is located in the north-east
corner of the control room. You will see a blue label above the console
that says "CAL/ICD" -- actually it appears that someone has stolen that
label, oh well.*

4.  **How do I reboot the D0OL45 node? Is that ok?**

*There is no problem rebooting the node in the sense that you do not
need to check with anyone. Assuming it has hung and is not responding,
just press the reset button on the computer which you will find
.........*

5.  **How do I log on as a d0run user (rather than d0cal)?**

*Easy. There are two ways depending on your need.*

a.  *If you don't need any GUI's, then in an xterm window simply type
    **su d0run**, you will then be prompted for a password which you can
    find in the DAQ shifter's bible under Group Accounts.*

b.  *If you need GUI's, then in an xterm window type **start_cal d0run**
    (which basically does **setup d0online** and then **d0ssh --l d0run
    d0ol45**)**.** Replace d0ol45 by whatever other machine you would
    like.*

# Window matters

1.  **How do I pop up a new window/shell?**

If other windows are open, do NOT open another window unnecessarily as
it wastes computer resources. Click the left-button on the mouse in any
open space on your window, then select ***New Shell*** to open an xterm
window. Except if you want to open histoscope, it is prudent to enter
***setup d0online*** once it is open -- it sets up what you will need,
and it never hurts to do that.

2.  **How do I close an X-window?**

*If you have tried the normal methods to get rid of it (like exit within
the window) and failed, then **left-click** your mouse in an **open area
on the desktop**. You will see a set of possible choices, select
**Window Operations** and then select **Close**. That produces a "skull
& crossbones" cursor which you can move to the window you want to close
and left click. That should close it.*

# Printing

1.  **How do I print a file?**

*In your xterm window type **printbw filename**, where **filename** is
the name of the file you would like to print. The output will come out
on the hp8150 printer that is located just outside the control room (as
you leave the control room area turn immediately to your left, and it is
right there). This is equivalent to* ***flpr --q dab1_hp8150_d
filename*** *The command **flpr --q dab1_hp8150 filename** yields a
single-sided printout.* *The command* ***print_color filename*** *will
print on the color printer.*

2.  **How do I just print what I see in an X-window?**

*From any window (after issuing the **setup d0online** command), you can
run the command **print** (carriage return), then left-mouse click on
the window that you would like printed (note that if you have an
overlapping window on top, it will print that piece of it as well!). The
output comes out on the HP 8000 printer just outside the control room.*

# Useful Tools

1.  []{#_Ref4211357 .anchor}**How can I capture a screen shot?**

*Well, this is UNIX, so while it can (of course/not only) be done, it is
(never/even) transparent, (but/and) there are many ways. Don't forget to
clean up after yourself rather than keep cluttering the disk area with
these screen captures. Here are some possible ways.*

a)  *[Making a gif image file]{.underline}: In an xterm window on the
    same desktop as the screen you want to capture, type **cd** to go to
    home directory then **import screen.gif**. The cursor changes to a
    small "**+**" symbol. Move the cursor to the window you would like
    to capture and then left-click in that window, and wait until it is
    done. Check that **screen.gif** is in the directory by typing **ll
    /home/screen.gif** (this is the absolute path to the file you
    created).*

b)  *[Making a jpg image file]{.underline}: Same as for gif image. Just
    change gif to jpg.*

c)  *[Making a ps image file]{.underline}: Same as for gif image. Just
    change gif to ps.*

d)  *[Alternate method]{.underline}: in a xterm window run **display**,
    then select **File**, followed by **Open** and then click on
    **Grab**, enter a value of about 5 seconds (enough time to get the
    cursor to the window you want to capture). Move the cursor to the
    window you want to capture and wait the 5 seconds until the cursor
    changes shape to a cross, then click in that window. An image of
    that window will now appear in the display window; left click on the
    image and select **File**, then **Save**. You can add an extension
    to the name to save it in that graphics format -- for example .gif
    or .jpg.*

e)  *[Yet another method]{.underline}: in an xterm window run **xwd
    \>filename.screen**, the cursor changes to a small "+" symbol. Move
    this cursor to the window you would like to capture and click on it.
    That will capture the screen. Now convert that file to jpg by typing
    **convert filename.screen filename.jpg**. That will produce the file
    **filename.jpg** in jpg format in the **/home** directory.*

*Now you need to be able to pick up this image file, so we normally **mv
filename \~/calwww/histos/** which is accessible from the web at
**www-d0online.fnal.gov** by then going to the group pages, selecting
the Cal page and looking at the directory of histogram files. Copy it to
wherever you want.*

f)  **How do I convert a postscript file to a jpeg I can insert in the
    e-log?**

*Easy. Once you have a postscript file (where for example you saved the
plot as a .ps file), then you can run the following script
/home/d0cal/plot4log filename.ps. This will create a jpeg file utilizing
ghostscript with the name filename.jpg. To insert that in the e-log,
follow the directions on page [64](#_Ref19011316).*

g)  **How do I edit a file in Unix?**

*If Unix is your OS, then you know, but for those Microsoft lackeys
among us, then one simple WYSIWYG tool is **xemacs filename &**. That
will pop up a separate window with a windows-like GUI editor. There are
surely many other ways, but this one seems easy. You may first have to
issue a **setup xemacs** command.*

h)  **The various GUI's are slowing down, how can I check to see if the
    system is overloaded?**

*You can check to see if the system is overloaded (for example lots of
lines in the GUI's are greyed out) by running a local copy of d0.ioc. To
do that run **start_cal ioc.** You are likely in trouble if **d0ctl03**
shows more than **70% CPU** or **70% MEM** usage. If it is urgent
(because the system has come to a grinding halt), then page the cal
expert. If it is just annoying but you are still running, then wait till
the daytime to page the expert.*

i)  **How do I check to see what account the open window belongs to?**

*Run **whoami** in that window.*

j)  **Where can I find the run summary information?**

*As long as you wait about 10 minutes after the end of a run, you can
access the run summary information by doing*

> *\> setup d0online*
>
> *\> runsum.py xxxxxx* (where xxxxxx is the run number)

k)  **How do I find a "rogue" node on my node?**

*This may be a process that you did not know was running. I need to get
the info/steps from Alan Stone -- this is a placeholder.*

## 

# Taker

1.  **What is a taker?**

*Taker refers to the actual data taking program. It may be running
because the DAQ shifter started a Global run, or you may be running one
when you take pedestals, download the trigger, etc.*

2.  **How do I open a taker?**

*In any xterm window (Monitor #0), type **start_cal** **taker**.* Be
patient, t*hat pops up a taker window.* You can check if recording is on
or off, by looking at the upper right hand corner of the taker window,
which will show ***Recording: on***. or ***Recording: off***. To change
the mode, left click ***Modify*** and check the ***Recording*** box (as
indicated by the little red box next to it).

3.  **How do I remove the Calorimeter crates from the Global Run?**

You should ask the permission of the Shift Captain to remove the
Calorimeter crates from the global run, and you should ask the DAQ
expert to remove them once you have approval.

4.  **The instructions say to run a specific trigger (with no version
    number), yet when I look in the list available in taker I see
    version numbers. Should I ignore the version number?**

*Yes. To make the documentation more robust and stable, we will give the
trigger file name **without a version number.** If there is only one
version, use it. If there are several versions, ask the calorimeter
expert which one to use. If he is not available, use the latest
version.*

5.  **How do I set the hardware parameters so that we are in data taking
    mode?**

*In the Taker, click on **modify**, and then click on **Change
Trigger**. You will now see the configuration window. To make sure you
are in the correct directory, click **UP (Left)** until you are all the
way "up" and the directories no longer change. Double click on the
directory **commissioning**, then double click on **cal**, then double
click on **cal_prepare_for_run-xxx (where xxx is the version number and
if you find more than one version call the on-call expert)**. It will
tell you **download in progress**. This will among other things download
the pedestals. When you are finished, make sure to free the trigger. You
do that by clicking **modify** in the Taker, and then selecting **Free
trigger**.*

# Electronic Logbook

1.  **How do I start the electronic logbook?s**

In an xterm opened in Monitor #0, ***type*** ***setup*** d0online and
***start_daq logbook.*** Once the e-log window pops up, left-click on
the ***DETECTOR Shift*** tab at the top, next left-click on the
***CALMUO*** heading in the menu bar at top, and left-click on the only
pull down menu entry, ***CALMUO Log***. Expand the CALMUO Log window
(left-click on the expand window symbol -- 2^nd^ from the right on the
top bar). Click on the ***Log In-Log Out*** button on the left and enter
your name and password.

2.  **How do I get a password for the electronic guidebook?**

*Ask one of the experts to create an account for you. The DAQ shifter or
Captain can also create an account for you. This is what they will do.
In the logbook click on **Administrator** in the left-hand menu. In that
menu click on **Select Administrator** and then select **Shift
Administrator**. The password is the same as the one for d0run which the
DAQ shifter and Captain should know. In the new window select **Add
User** and fill in your information.*

3.  **How do I add text to the electronic logbook?**

Open a text window by left-clicking (and holding the left mouse key
depressed) on the ***Text*** button (top right of screen), then drag and
drop (i.e., release the mouse button you have been holding depressed)
the little paper symbol onto the main screen of the CAL log window. That
will open a text window at the bottom, where you can start entering your
comments. When you are done with an entry, you can save it by
"archiving" it (right-click in the text window and select ***Archive All
non-archived Entries in the Topic***, which will save the entry with
your username and time stamp (so you don't need to enter that info in
your entry). If you want all the gory details, there is a full users
manual that you should find floating around the control room.

4.  **I would like to go back and edit an archived entry -- how do I do
    that?**

*Sorry, as far as I can tell you are screwed (at least for the moment).
What you can do is annotate an entry to make a correction or comment
after the fact. To annotate, just left click the **annotate button**
that is attached to each archived entry.*

5.  **How do I print an entry I just made in the logbook?**

*This is strange but true -- you can't do this easily, despite the fact
that you will find menu items that should let you do this (click on
**Entries**, then select **Print Selected Entries** but you can't
actually check the printer box -- sigh). Here are two possible
workarounds: (a) print to a file (rather than printer) following the
above steps, then try and print that somehow, or (b) after having
archived your entry, then use the web based tools to search for the
entry from* <http://www-d0ol.fnal.gov/crl/d0search.jsp>*, and when you
actually find the entry of interest use your browser print function to
print. Simple, right.*

6.  **I know there is an old logbook entry that I want to see -- how do
    I find it?**

*One way is to use a web browser and go to*
[http://www-d0online.fnal.gov/crl/
Jsp2.jsp?inquiry=d0cal/72hours](http://www-d0online.fnal.gov/crl/ Jsp2.jsp?inquiry=d0cal/72hours)
*where you will find links which will let you access the last 72 hours,
the last week or the last two weeks of log entries.*

*You probably don't need this, but here are more detailed instructions.
It is probably me, but I don't find the e-log user transparent. So this
may not be the approved method but it seems to work. On the left side of
the e-log window you will find a **Search** button -- left click on it.
That opens up the search window. Use the **Inquiries** tab if you have
specific search criteria you want to use (they are more or less self
evident), or use the **Log Entry Explorer** tab if you are just fishing
for all entries on a particular day or time. You then **Execute
Inquiry** by left clicking on the button, and after awhile you should
see the button below that change from **Found 0** to **Found 10** (for
example if it finds 10 entries that satisfy your search criteria). If
you hover on that button it will say **Drag this to Report Container**.
If you are like me you'll ask what the hell is a Report Container? That
cryptic statement means that you open another **CAL log** by left
clicking on the **CAL** tab (upper left) and selecting **CAL Log**, that
in turn opens a **CAL Log** window (this IS a Report Container, I
guess). Now left click and hold the mouse on the **Found xx button** and
drag-and-drop it into that window you just created. The entries will now
appear there.*

7.  []{#_Ref19011316 .anchor}**How do I import a screenshot or jpeg file
    into the logbook?**

*Let's assume that you already have a jpeg file of the screenshot in
some directory (if you don't know how to get a screen capture just look
at page [59](#_Ref4211357)) and that you have an open logbook entry into
which you would like to paste it. Now go to the logbook and look for the
**Insert Image** menu at the top of the logbook window and click on it
and select **Insert Image from a File**. That will pop up a dialog box
asking you to enter the directory and filename. If you don't know the
name, you can **Browse** -- usually it is in the **/home** directory
(you can get to the home directory by clicking on the icon that looks
like a house/home -- get it?). Once you have entered the filename click
**OK** in the dialog box. That will import the jpg image into the
logbook.*

8.  **How do I copy and paste text from an xterm window into a logbook
    entry?**

*In the xterm window which has the text you wish to copy, highlight the
text using your mouse (position the mouse at the start of the text, hold
the left mouse key depressed, drag it across the text you want to copy
and release the mouse). Leave that text selected and move the cursor
into the logbook window. Left-click on the **edit** pull down menu and
select **paste-from-clipboard**. It will copy the selected text to the
logbook at the location of your cursor. This does not appear to work all
the time! (ah, Unix).*

# GUI's: generic how to

1.  **How do I start the standard set of calorimeter monitoring GUI?**

The standard GUI's should be running on Monitor #1 and #2. The basic
GUI's required to monitor the health of the calorimeter system can be
started by typing ***start_cal shifter*** in any xterm window (no setup
d0online is necessary).

2.  **How can I remember the name of a GUI?**

If you want to know the GUI names, then run ***start_cal*** with no
name, which will list the available names.

3.  **How do I start a particular GUI?**

You can start it up by itself by running ***start_cal \<name\>***, where
***name*** is the GUI you want to start, for example ***alarm*** or
***supply*** . See above if you don't know its name.

# Supply GUI's

1.  **What does the VBD reset in the Calorimeter Power Supply Monitor
    Display GUI (= Supply GUI) do?**

*In the Calorimeter Power Supply Monitor Display GUI (= Supply GUI) you
may have noticed three buttons that are labeled T&C reset, ADC reset,
and **VBD reset.** Together with the SCL init, these are the various
resets that can be issued to the calorimeter system. The **VBD reset**
is actually an artifact of the system which **no longer does anything**.
It was supposed to send a reset to the crate readout board (called a VBD
in run 1, used early in run 2, but now replaced by an SBC). If it were
implemented, it should only be pushed when the DAQ system is NOT running
with ANY CAL crates.*

2.  **What does the ADC reset in the Calorimeter Power Supply Monitor
    Display GUI (= Supply GUI) do?**

*In the Calorimeter Power Supply Monitor Display GUI (= Supply GUI) you
may have noticed three buttons that are labeled T&C reset, **ADC
reset**, and VBD reset. Together with the SCL init, these are the
various resets that can be issued to the calorimeter system. The **ADC
reset** command tells the ADC board to abort any operation in progress
(including the handshake with the VBD/SBC which if underway will provide
corrupt data to the VBD/SBC), and return to its initial power on state.
Most (but not all) of the resets are preformed at the beginning of every
ADC digitization cycle, so this should never be required. However, there
are states that the ADC crate can get into, where it will not recognize
a new Start Digitize cycle, and the only way to fix it is to send this
reset. It should only be done when the DAQ system is NOT running on ANY
CAL crates. Make sure the run is paused before you issue such a reset.*

3.  **What does the T&C reset in the Calorimeter Power Supply Monitor
    Display GUI (= Supply GUI) do?**

*In the power Calorimeter Power Supply Monitor Display GUI (= Supply
GUI) you may have noticed three button that are labeled **T&C reset**,
ADC reset, and VBD reset. Together with the SCL init, these are the
various resets that can be issued to the calorimeter system. VBD reset.
The **T&C reset** command is ORed with the SCL init at the earliest
possible place on the board, so it should function exactly like an
SCLint, with the following exceptions. It is issued asynchronized to the
accelerator clock (which should not make any difference, except that the
different cards will then be using different SCA cells for the same
physics event). It works even when the SCL link to the trigger system is
down. Since we usually only run the system when the trigger system is
working, this feature is only useful for the expert. So there is no
reason that this reset should ever be needed. But if used, it should
only be done when the DAQ system is NOT running with ANY CAL crates.
Make sure the run is paused before issuing this reset.*

4.  **What is the Channel Archiver?**

*The Channel Archiver is a program that collects and archives all the
voltages, currents, etc every 15 min. There are programs that then allow
you to look and plot variable values that are stored in the archive.*

5.  **How do I restart the channel archiver?**

*For various (and sometimes mysterious) reasons, the channel archiver
process may stop running and archiving -- in that case all that
monitoring information is lost. If you see that it is stopped you can
restart it as follows. Note that the directory file name should be
included to prevent a new archive from being created. Open an xterm
window and move to the proper directory by typing **cd
/projects/archive/cal/current**. If the lock file is present (to get a
listing of files, you could type **ls**), then you must remove it by
typing **rm archive_active.lck** Now restart the archive but use the
directory name as one of the command line parameters. For example if the
directory name is **dir.20011019-091652**, then type
**start_chan_archiver --c config_file --p 4814 --w . --d
dir.20011019-091652**.*

# How to grab the file before SAM gets it

Since getting data files back out of SAM can be problematic, here is a
method that you can use to grab the file before SAM whisks it away.

1.  Log into d0olc. Use ssh to login with d0***ssh --l d0cal d0olc***

2.  

3.  Find the file. Change directory, ***cd /buffer***, then list the
    files and find their directory using ***ls --l \*/\*x\*.raw*** where
    *x* is the run number.

4.  Copy the file. Once you have found the directory, you can copy the
    file to the project disk ***/projects/D0cal/data***. For example,
    the command to copy is ***cp buf015/daq_test_0000129054\*.raw
    /projects/D0cal/data/.*** (yes, with a point after the last slash.
    You should make sure that the file is not too big (no more than
    about 1000 events).

# Pulser GUI's

1.  **How to shutdown all pulsers?**

If you need to shutdown the pulsers, then you are likely in big trouble
because all your previous hard work is probably garbage. So turn off the
pulsers and call an expert. To check the state of the pulsers, go to the
pls mode tab in the ***Calorimeter Power Supply Monitor Display GUI (=
Supply GUI)*** and see that all entries show zero. If not, the to turn
them off, open the pulser control GUI. If it is not running, run
***start_cal pulser*** in your xterm window. Once the GUI starts, shut
down all the pulsers by clicking the ***Select All*** button and
clicking the ***Shutdown*** button.

2.  **How do I check that the ICD pulser is off?**

*There are two ways to do this.*

*[The Easy Way]{.underline}: Look in the **CalorimeterPower Supply
Monitor Display GUI (= Supply GUI)** and select the **PLS MODE** tab.
The last row is entitled **ICD LED** and contains the information on the
delays and voltages. See that they are both set to **0**. You can use
the next method to change it if you need to.*

*[The Hard Way]{.underline}: You essentially follow the procedure for
setting the ICD pulser. So, to start, **open an xterm window** in the
usual manner (see above). Then type **setup d0online**, and change the
directory by typing **cd /home/d0icd/vme/** and then run the ICD LMB
interface by typing **./lmb_int.py &**. That pops up a small GUI. In the
All column type in **0.0** for the **Voltage** and **0** for the
**Delay** (which may already be there), then click on the **Download**
button. That will assure you that the pulser is off by setting all the
values to zero.*

# Control Software

1.  **Should I, as a shifter, issue any kind of reset?**

*No. The only thing you should do is ask the DAQ shifter to issue an SCL
Init after the run is paused. Otherwise leave it to the experts.*

3.  **What is an SCL init?**

*SCL stands for Serial Command Link (SCL). It is the signal (sent on a
fiber) that contains all the commands from the trigger framework for the
calorimeter system (and other systems). You may hear that the DAQ
shifter is issuing an SCL init -- that resets the signal to all systems.
Nowadays we are more cautious than in the past, so you should NEVER
issue one yourself during global running (so I won't tell you how!), but
rather ask the DAQ shifter to pause the run and to issue one if you need
it.*

4.  **When should I issue T&C, ADC or VBD resets vs an SCL Init?**

As a shifter, the short answer is NEVER. The reasons are as follows. In
principle, the only reset that should ever be needed is an SCLinit.
However, in the past (and maybe even still true) we have observed that a
SCLinit does not clear the system properly, so additional reset points
which exist in the hardware have been made available. There are a few
\"features\" of the SCLinit which make it the \"preferred\" way to
recover from an error, assuming that it works. First, it has been
designed to be able to run \"without pausing the DAQ system\". ALL the
local resets available in the CAL Gui\'s will definitely cause problems
if used when the DAQ system is \"running\". Most, but not all of these
errors will automatically be fixed over the next few events, but as a
general rule one should always pause any runs using the CAL crates
before issuing the local reset. Second, it occurs in ALL crates over the
entire DAQ system (not just CAL), on exactly the same 132ns clock cycle.
The local resets exist on individual crate level, as well as grouped for
all CAL crates, and are guarrenteed NOT to occur at exactly the same
time in any two CAL crates. Third, the SCLinit does not actually reset
everything. The ADC and \"VBD\" were designed to never hang, so should
never require a reset. Bottom line: leave those other resets to the
experts and ask the DAQ shifter to issue an SCL Init when the run is
paused.

Reset a tripped low voltage supply. If a power supply has tripped (which
normally means that there are a large number of pink boxes), reset it by
clicking the ***Reset*** button corresponding to the supply that is off.
If all supplies have tripped, then use the ***Reset all*** button.
Sometimes it takes more than one click to reset then try resetting it.
If there is only a single isolated pink box (rather than the whole row
indicating the supply might have tripped) which is not in the reference
plot, then resetting should normally be useless. First check the
Troubleshooting guide on page [87](#_Ref15896968), but if that fails,
then call an expert. While you are waiting for the expert, if the power
supply problem is with the BLS supplies, then you should advise the DAQ
expert that the corresponding crate be taken out of the run (because the
reconstruction will go nuts if it uses these data). However, you
eventually want to remove only the minimal number of channels -- so once
you have identified the effected channels, you should "kill" those
channels. After the bad channels are killed, the crate can be put back
in the run. Use the ***Hot Cell Killer*** to suppress them. Start the
hot cell killer by typing ***setup d0online***, then ***d0ssh
d0run@d0ol45***, then ***cd /online/comics/cal***, and finally
***NEW_HotCellKiller.py***. Check the help menu for usage details.

# Monitoring

1.  **Starting dq_calo**

In any xterm window logged on d0ol45 as d0cal, type ***start_cal
dq_calo***. You will see some lines written to the screen. After a
minute or so, check that dq_calo started by typing ***ps -C
dq_calo_x***. If an instance of dq_calo is already running, nothing will
be done.

2.  **Starting dq_monitor**

In a xterm window of Monitor #0 terminal logged on d0ol45 as d0cal, type
***start_cal dq_monitor***. A GUI should appear.

Here is a view of the calorimeter that you may find useful. It lists the
layers (and their names) in the radial direction, plotted against the
towers in the direction.

![](media/image1.png){width="4.802777777777778in"
height="6.742361111111111in"}

Figure 2

  --------------------------- ------ ------------------------------ ------
                                                                    

  Cal Examine                        Preamp Low Voltage Power       
                                     Supplies                       

  Taker                              BLS Low Voltage Power Supplies 

  Calorimeter Power Supply           High Voltage                   
  Monitor Display GUI                                               

  Histoscope                         Calorimeter Workstation        

  Data Taking Problems                                              

  Calib Problems                                                    

  Alarm Server/Significant                                          
  Event Server (SES)                                                

  Downloading & Creating                                            
  Pedestals                                                         

                                                                    

                                                                    
  --------------------------- ------ ------------------------------ ------

# 

# 

1.  
2.  
3.  
4.  
5.  

# 

1.  
2.  
3.  
4.  
5.  
6.  
7.  

# 

1.  
2.  
3.  
4.  
5.  

# 

1.  
2.  

# 

1.  
2.  
3.  

## 

1.  

# 

2.  
3.  

# 

4.  
5.  
6.  

## 

1.  
2.  
3.  

## 

1.  

## 

1.  
2.  
3.  

## 

1.  
2.  

# 

## 

1.  

# 

1.  

# 

# 

2.  

# 

3.  
4.  
5.  
6.  

# 

1.  
2.  
3.  

## 

1.  

# 

[]{#shifter_task_chapter .anchor}Expert Pages

In this chapter we present a manual for tasks that are normally done by
experts, not shifters. On occasion you may be called upon by the experts
to perform some of these tasks

# Calorimeter Experts

+---------------+------------+------------+------------+------------+
| Name          | Office     | Home phone | Pager      | Cell phone |
|               | phone      |            |            |            |
+---------------+------------+------------+------------+------------+
| Mike Arov     | \(630\)    |            | \(630\)    | \(815\)    |
|               | 840-8131   |            | 840-0935   | 501-1861   |
+---------------+------------+------------+------------+------------+
| Ashish Kumar  | \(630\)    |            |            |            |
|               | 840-8595   |            |            |            |
+---------------+------------+------------+------------+------------+
| Silke Nelson  | \(630\)    | \(773\)    |            | \(773\)    |
|               | 840-8301   | 645-0021   |            | 209-3039   |
+---------------+------------+------------+------------+------------+
| Nirmalya      | \(630\)    | \(630\)    | \(630\)    |            |
| Parua         | 840-6792   | 985-1160   | 218-8489   |            |
+---------------+------------+------------+------------+------------+
| Pierre        | \(630\)    | \(630\)    |            |            |
| Pétroff       | 840-6447   | 840-3571   |            |            |
+---------------+------------+------------+------------+------------+
| Kirti Ranjan  | \(630\)    |            |            |            |
|               | 840-8595   |            |            |            |
+---------------+------------+------------+------------+------------+
| Dean          | \(630\)    | \(631\)    |            |            |
| Schamberger   | 840-2495   | 689-3141   |            |            |
|               | (631)      |            |            |            |
|               | 632-8094   |            |            |            |
|               | (631)      |            |            |            |
|               | 632-9221   |            |            |            |
|               |            |            |            |            |
|               | \(631\)    |            |            |            |
|               | 632-8084   |            |            |            |
+---------------+------------+------------+------------+------------+
| Junjie Zhu    | \(630\)    | \(630\)    |            | \(630\)    |
|               | 840-5252   | 740-4533   |            | 740-4533   |
+---------------+------------+------------+------------+------------+
| Robert Zitoun | \(630\)    | \(312\)    | \(630\)    | \(630\)    |
|               | 840-2694   | 341-1696   | 266-0634   | 399-0024   |
+---------------+------------+------------+------------+------------+

# 

# ICD Experts

  ---------------- ------------- ------------- ------------- -------------
  Name             Office phone  Home phone    Pager         Cell phone

  Lee Sawyer       \(318\)                                   
                   257-4053                                  

  **Andy White**   **(817)                                   
                   272-2812**                                
  ---------------- ------------- ------------- ------------- -------------

#  Trigger Experts

  ---------------- ------------- ------------- ------------- -------------
  Name             Office phone  Home phone    Pager         Cell phone

  Dan Edmunds                                                

  Bob Kehoe        \(630\)       \(630\)       \(630\)       
                   840-8653      762-8549      218-4801      

  **Joe            **(630)       **(312) 421   **(630)       
  Kozminski**      840-3983**    1629**        218-4462**    
  ---------------- ------------- ------------- ------------- -------------

# Other Experts

  ---------------- ------------------ ------------- ----------------------- -------------------
  Name             Office phone       Home phone    Pager                   Cell phone

  Ursula Bassler   \(630\) 840-8740   \(630\)       <0612734678@sfr.net>,   011-33-612-734678
                                      840-4230                              

  Laurent Duflot   \(630\) 840-6852                                         

  Shaoha Fu        \(630\) 840-8757   \(630\)                               
                                      840-4923                              

  Leslie Groer     \(630\) 840-5587   \(630\)       \(630\) 218-8421        
                                      208-8151                              

  Slava Shary                                                               

  Michael Tuts     \(630\) 840-8304   \(212\)                               \(917\) 627-4646
                   (914) 591-2810     316-1902                              
                   (212) 854-3263                                           

  Christian        49-6131-392-3668                                         
  Zeitnitz                                                                  
  ---------------- ------------------ ------------- ----------------------- -------------------

# 

# Check List

1.  Information. When you get the pager, exchange information with the
    previous expert. At the end of your shift, send a **concise
    summary** of what happened during your shift (calibration runs,
    physics run...) to the expert list. State the problems you faced and
    the way you solved them.

2.  Make sure pedestal calibration run are taken. Pedestal calibration
    runs must be taken regularly (\<t\> \~ 1day). Make sure a
    calibration has been taken recently, otherwise remember the shifter
    to take one at the end of the next store.

3.  Make sure a pulser run is taken.

4.  Make sure preamp temperature are monitored.rs which by convention we
    call **Monitor #0** (bom left), **Monitor #2** (top left)t each mon
    workspace in the t hand corll refer to in tng section

Pulser Run

There are many times when you will want to take a very simple pulser run
just to see that everything is alive. For example, after an extended
shutdown when there has been lots of activity in the detector and cards
have been swapped, it would be prudent to carry out such a run just to
check that all channels are basically OK. Experts may ask you to run
different parameters, but lacking any further guidance, use these
values.

1.  Set the configuration. In the taker, select ***Modify Trigger*** and
    download file ***commissioning/cal/pulser-cal-simple***.

2.  Verify or modify the settings.

3.  Start the Pulser run. Click on ***Start*** to begin the run. A
    pop-up ***Begin Run*** window will appear. Fill it in with your name
    in the ***shifter*** box, the gain (x1 or x8) and the crates being
    run in the ***Reason*** box, and check the ***Calibration*** button
    (or ***Test*** if you don't want it saved). That box disappears and
    the run has started. 500 events take about 5 minutes.

4.  Stop the Run. When all the data (500 events) has been collected, you
    should stop the run by clicking ***Stop*** on the Taker. Write the
    run number in the logbook with any other relevant information.

# Delay Ramp Run

This set of runs will allow you to generate data for a series of fixed
pulser signals where the time delay of the pulser signal relative to a
fixed sampling time is varied in programmable steps.

1.  Set the timing for baseline sampling. The timing for the baseline
    sampling is set by opening the CRATE MONITORING GUI (=T&C GUI),
    clicking on the ***Global BLS*** ***Mode*** button and clicking on
    the ***x1 --2 ticks*** button. If it isn't open, go to your xterm
    window and run ***start_cal tandc***. Don't forget to always close
    the pull down menus before proceeding or you may be sorry when you
    forget where the menu came from and you can't close it! You have now
    set the baseline to be sampled at --5BC rather than the normal
    --3BC. The gain is fixed at x1 (which should be used for pulser DAC
    settings of \>10000).

2.  Set the ADC mode. Click on the ***Global ADC Mode*** button in the
    CRATE MONITORING GUI (=T&C GUI) and select ***Ped Sub***. This will
    read out all cells with pedestals subtracted but without zero
    suppression.

3.  Set the T&C mode. On the ***CRATE MONITORING GUI (=T&C GUI)***
    (called Crate Monitoring on the GUI) click on the purple ***Global
    T&C Set*** button and select ***Si Buff Normal*** (which stands for
    single buffer normal). In an xterm window change the directory,
    ***cd /online/config/cal/vme***, execute the usual ***setup
    d0online***, and then run ***./vme_tc_ctrl &***. Now select (left
    click and hold while you sweep over the entry with the mouse cursor)
    the number in the 1^st^ column and type in ***89***. Similarly, for
    the 4^th^ column, type in ***1f0***, and put ***8000*** in the 7^th^
    column. These values should now be in the boxes (with a 0x in front
    of them). Note that the number in front of the ***f0*** determines
    the number of ticks early that the ramp will be (so ***n***f0 = ramp
    ***n*** ticks early or late, and ***n*** goes from ***0 to 7,***
    where late starts about something like 4 \[Editors note: the exact
    transition between early and late needs to be checked\])). You are
    done, so ***Quit***. You also need to enable Trigger ***Bit 12***
    (Fixed Cell) in the Pulser GUI (you will find it near the bottom
    right) -- remember to start the pulser GUI you can type ***start_cal
    pulser*** in an xterm window (after a ***setup d0online*** of
    course).

4.  Verify the T&C setting. Open the Calorimeter Power Supply Monitor
    Display GUI (= Supply GUI) (if you don't remember how to do that, go
    back and look at the beginning of this chapter), and click on the
    ***Mode*** tab (the last one on the right). The value of ***DIAG***
    (in the last orange/brown column) depends on the value of the
    ***TICK*** setting you made earlier. If ***TICK***=0 then
    ***DIAG***=0. If ***TICK*** is not equal to zero, then ***DIAG***
    should not be equal to zero. The value of ***MODE*** should be
    ***0x8a09*** if you selected ***x1 -- 2*** ticks , or ***0x8b09***
    if you selected ***x8 --2*** ticks.

5.  Set the initial pulser configuration. This step will ensure that the
    pulsers are in a known starting state. Open the pulser control GUI.
    If it is not running, run ***start_cal pulser*** in your xterm
    window. Once the GUI starts, shut down all the pulsers by clicking
    the ***Selecting All*** button and clicking the ***Shutdown***
    button. Now initialize the pulser settings in the GUI by selecting
    the ***All*** button for the Pulser Address and then deselecting
    address 12 (usually the ICD is left out). Note you may need to click
    this twice to actually set it and other buttons in this GUI. Now set
    the DAC value to ***25000*** (for x1) or ***5000*** (for x8). Click
    the ***Enable All*** button to select all channels. Click the
    ***Cmd*** All button to set all the ***Cmd*** buttons. Set the
    ***Delay All*** box to the initial value of ***0***. Finally click
    the ***Download!*** Button.

6.  Check the Pulser settings. Open the Calorimeter Power Supply Monitor
    Display GUI (= Supply GUI) and go to the ***PLS mode*** tab. Make
    sure that the selected channels for all selected pulsers show a 1.
    The selected commands for all selected pulsers should be set to 1.
    The DAC values for the selected pulsers should be set at the chosen
    value (25000 if you have been following these directions). The six
    delay values for all the pulsers should be set at the chosen initial
    value. All the non-selected pulsers (none if you have been following
    these instructions) should have DAC values of 0, Delay values of 0,
    and channels and commands values of 0.

7.  Monitor the ramp with cal_elec_ramp. You should monitor the status
    of your run using Cal Elec running in ramp mode. That means starting
    ***cal_elec_ramp***. See the detailed instructions on page 157 on
    how to do that.

8.  Start the Delay ramp in the PIB (Pulser interface Board). Go back to
    the pulser GUI and click the ***Delay Ramp*** button -- the
    ***Request*** box in the upper right should show a value of **66**.
    Next enter the ramp parameters: ***Steps = 50*** (max 255); ***Step
    Size = 5*** (min 1, max 255); and ***Ev/Steps = 50***. Now click
    ***Download!*** (only once!) to send the ramp request to the PIB.
    Note that if the initial value in the Current Ramp Status pop-up
    window is not zero, then the previous ramp was not stopped. In that
    case stop the ramp in the Ramp Status Window.

9.  Start the Pulser run. Open a Taker and select ***Modify Trigger***.
    Select the ***pulsar-cal-ramp-1.3*** file in the commissioning/cal
    directory. Click on the taker ***Start*** button. You should see the
    events changing in the ramp status window. Note that if the runs
    gets interrupted for some reason and you have to start all over
    again, then make sure to stop the ramp in the ***Ramp Status
    Window***. If you have trouble, check with the DAQ shifter and
    verify that there are no runs either underway or downloaded with
    zero L3-nodes associated. If that is the case, then ask him/her to
    stop the run, free the trigger and restart with a fixed (non-zero)
    number of L3 nodes assigned to their runs.

10. Stop the Run. When all the data has been collected (as verified in
    the ramp status window), you should stop the run by clicking
    ***Stop*** on the Taker. You should also stop the ramp in the Ramp
    Status Window. Write the run number in the logbook with any other
    relevant information. Send a copy of that to Ursula at
    [mailto:bassler@in2p3.fr](mailto:bassler@in2p3.fr)

  ----------- ------------ ------- ------------- -------------------------
                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 
  ----------- ------------ ------- ------------- -------------------------

## 

## 

## 

## 

##  {#section-39 .unnumbered}

1.  
2.  
3.  
4.  

## 

5.  

## 

## 

##  {#section-43 .unnumbered}

1.  
2.  

## 

## 

## 

3.  

## 

## DAC Ramp Run

## 

11. 
12. 
13. 
14. 
15. 
16. 
17. 
18. 
19. 
20. 
21. 

## 

## This set of runs is designed to measure the linearity using the pulser at different settings.

1.  

2.  

3.  

4.  

5.  

6.  

7.  

8.  

9.  

10. Set the run parameters. In the taker, select ***Modify Trigger***
    and download file ***commissioning/cal/pulser-cal-DAC-rampx8 (or
    x1)***.

11. Check the Pulser settings. Open the Calorimeter Power Supply Monitor
    Display GUI (= Supply GUI) and go to the ***PLS mode*** tab. Make
    sure that the selected channels for all selected pulsers show a 1.
    The selected commands for all selected pulsers should be set to 1.
    The DAC values for the selected pulsers should be set at the chosen
    value (0 if you have been following these directions). The six delay
    values for all the pulsers should be set at the chosen initial
    value. All the non-selected pulsers (none if you have been following
    these instructions) should have DAC values of 0, Delay values of 0,
    and channels and commands values of 0.

12. Start the run. Click on the taker ***Start*** button. You should see
    the events changing in the ramp status window. When a new ramp step
    starts, the delay values in the supply-all GUI will change. Note
    that if the runs gets interrupted for some reason and you have to
    start all over again, then make sure to stop the ramp in the Ramp
    Status Window. If you have trouble, check with the DAQ shifter and
    verify that there are no runs either underway or downloaded with
    zero L3-nodes associated. If that is the case, then ask him/her to
    stop the run, free the trigger and restart with a fixed (non-zero)
    number of L3 nodes assigned to their runs.

13. Stop the Run. When all the data has been collected (as verified in
    the ramp status window), you should stop the run by clicking
    ***Stop*** on the Taker. You should also stop the ramp in the Ramp
    Status Window. Write the run number in the logbook with any other
    relevant information. Send a copy of that to Ursula at
    [mailto:bassler@in2p3.fr](mailto:bassler@in2p3.fr)

## 

## 

5.  
6.  
7.  

## 

## 

## 

## 

8.  

## 

## The following table is a quick reference. 

  ----------------------------------- -----------------------------------
                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      

                                      
  ----------------------------------- -----------------------------------

# DAC + Pattern Ramp Run

This type of run will cycle through various pulser patterns but differs
from the previous section by different steps for the x8 mode and a x1
mode as well. The following table is a quick reference. Notice that the
"Value-Value-Value" column in the table refers to the fact that you will
run the same cycle three times, and change the values as shown in the
table -- to make it clearer those values that change from run to run are
shown in bold. This whole cycle takes about 30 minutes.

  -------------------------- --------------------------------------------
  GUI/Parameter              Value -- Value - Value

  *In the T&C GUI*           

  **T&C Mode**               *Si Buffer Normal - Si Buffer Normal - Si
                             Buffer Normal*

  **BLS Mode**               ***X8-2ticks -*** ***x8-2ticks -***
                             ***x1-2ticks***

  **ADC Mode**               *Ped sub - Ped sub - Ped sub*

  *In the Pulser GUI*        

  **Pulser address**         Enable *All -* Enable *All -* Enable *All*

  **DAC value**              ***0 - 0 - 0***

  **Delay value**            *100* enter - *100* enter - *100* enter

  **Trigger bits**           Enable *13* *-* Enable *13* *-* Enable *13*

  *In Pulser GUI click on    
  "Pattern Ramp" to open sub 
  GUI*                       

  **Read File**              RunIItestpattern

  **Apply**                  click -- click - click

  **Close**                  click -- click - click

  *In Pulser GUI click on    
  "DAC Ramp" and enter*      

  **Steps**                  ***40*** ***-*** ***40*** ***-*** ***40***

  **Step size**              ***40 -*** ***400 -*** ***3200***

  **Ev/step**                *50 - 50* *--* *50*

  In Pulser GUI click on     
  "DOWNLOAD"                 

  **Power Supply GUI**       Check settings on PLS-MODE page

  *In Taker*                 

  **Modify**                 Recording *on*

  **Select a trigger**       *Change Trigger*

                             *Commissioning/cal/cal-all-xx*

                             *Download*

  **Start run**              

  Stop taker when ramp is    
  finished                   
  -------------------------- --------------------------------------------

It is assumed that you downloaded ***cal_prepare_for_run***. You have to
take 3 runs, a low DAC x8 run, a "normal" x8 run and a x1 run.

1.  Set the mode. In the Crate Monitoring GUI (T&C GUI),

    a.  click on the purple ***Global T&C Set*** button and select ***Si
        Buff Normal***.

    b.  click on the purple ***Global BLS Mode*** button and select
        ***x8-2ticks***.

    c.  click on the ***Global ADC Mode*** button and select
        ***PedSub***.

2.  In the pulser GUI.

    a.  Select the ***All*** button for the ***Pulser Address***. Note
        you may need to click this twice to actually set it and other
        buttons in this GUI.

    b.  Select trigger bits ***13***.

    c.  Set the ***Delay All*** box to a value of ***100*** (make sure
        you hit carriage return for this value to be entered).

    d.  Set the ADC value to 0.

    e.  Click on the ***Pattern Ramp*** button (near the middle-right of
        the screen), that pops up a new window. In this window, click
        ***Read File***, find file ***RunIItest*** pattern, then
        ***Apply*** and finally ***Close***.

    f.  Click the ***DAC Ramp*** button -- the ***Request*** box in the
        upper right should show a value of **65**.

    g.  Enter the ramp parameters: ***Steps =*** ***40***; ***Step Size
        =*** ***40***; and ***Ev/Steps =*** ***50***.

    h.  Click ***Download!*** (only once!) to send the ramp request to
        the PIB. If the initial value in the Current Ramp Status pop-up
        window is not zero, then the previous ramp was not stopped. In
        that case stop the ramp in the Ramp Status Window.

3.  Check the settings. In the Power Supply GUI, check the settings on
    the PLS-MODE tab.

4.  Start the low DAC x8 run. In the taker, select ***Modify Trigger***,
    then select ***cal-all-***xx in the ***commissioning/cal***
    directory. Click on the taker ***Start*** button. You should see the
    events changing in the ramp status window. Note that if the runs
    gets interrupted for some reason and you have to start all over
    again, then make sure to stop the ramp in the ***Ramp Status
    Window***. If you have trouble, check with the DAQ shifter and
    verify that there are no runs either underway or downloaded with
    zero L3-nodes associated. If that is the case, then ask him/her to
    stop the run, free the trigger and restart with a fixed (non-zero)
    number of L3 nodes assigned to their runs.

5.  Watch the changing patterns. As the data is being collected, you can
    open the Calorimeter Power Supply Monitor Display GUI (= Supply GUI)
    to the PLSMode tab where you will see the pulser patterns change.
    Make sure they are changing, otherwise something is wrong.

6.  Stop the Run. When all the data has been collected (as verified in
    the ramp status window), you should stop the run by clicking
    ***Stop*** on the Taker. You should also stop the ramp in the Ramp
    Status Window. Write the run number in the logbook with any other
    relevant information. Now repeat the process with new values in the
    following steps.

7.  Start the "normal" x8 run. Start everything from step 2e above, just
    change ***Step Size=400***.

8.  Start the x1 run. First set the BLS Mode to ***x1-2 Tick***. Then
    start again from step 2e above just changing ***Step Size=3200***.

Well, if you got this far it is now eight hours later -- go home, have a
drink and relax. You deserve it!

## 

## 

  -------------------------- --------------------------------------------
                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             
  -------------------------- --------------------------------------------

Otherwise, if you want the verbose directions to take such a run, follow
these directions.

9.  
10. 
11. 
12. 
13. 
14. 
15. 
16. 
17. 
18. 
19. 
20. 
21. 
22. 
23. 
24. 
25. 
26. 
27. 
28. 
29. 
30. 
31. 
32. 

Double Digitization Run

These runs take data simultaneously for the x1 and the x8 gain paths
separately. The basic procedure is the following

6.  Select the Taker . In the Taker, download file
    ***commissioning/cal/cal-double-digit-nopulser-xxx***. After the
    download is complete (about 30 seconds), ***start*** the run.

7.  Stop the Run. When all the data has been collected, you should stop
    the run by clicking ***Stop*** on the Taker. Write the run number in
    the logbook with any other relevant information.

# Triple Digitization Run

These runs take data simultaneously for three samples in time, which are
then used to carry out timing studies. Usually you would take these runs
while there is beam.

4.  Select the Taker. In the Taker, download file
    ***commissioning/cal/cal-triple-xxx*.** Set ***Recording off***

5.  Adjust the prescale. You need to adjust the prescale so that you get
    the highest rate without ADC errors. Ask the DAQ shifter what the
    ***and-or rate*** is for your trigger (it should be the only
    trigger). ***Divide*** that number ***by 4*** and use that as your
    prescale factor. (this should correspond to about a prescale of 25
    at a luminosity of 2E30). To set the prescale, click on
    ***Modify***, and then click ***Change Prescales...***, then enter
    the value of the prescale (e.g. 25) in the appropriate box (it
    should be obvious -- but ask the DAQ shifter if you need help). Now
    verify that the prescale is correct by clicking Start. The rate seen
    at the bottom of the window should be about 10-12 Hz (slower than
    that and you are wasting time, and faster than that and you will
    incur ADC errors). If it isn't in that range, adjust the prescale
    accordingly. Once you are satisfied, ***Stop*** the run (click on
    the bottom left button).

6.  Turn on recording and start the run.

7.  Check that there are not too many ADC errors. You should make sure
    that you are not getting too many ADC errors by watching the Mode
    status page (purple columns -- Status) 0x3f is normal but bits on in
    the high order 8-bit word indicates some error.

8.  Stop the Run.

# ICD LED Pulser Run

## 

1.  
2.  
3.  
4.  
5.  
6.  
7.  
8.  

## 

## The ICD needs to establish the running conditions (intensity and delay) for the LED pulser runs. In order to do this, we require a large set of runs taken under varying settings. Follow these instructions to take these sets of runs. You may wish to check in the logbook to see where we are up to. One run at a given voltage and delay setting will take about 30 minutes, and it should be done when there is no beam in the machine.

1.  

2.  

3.  

4.  

5.  

6.  

7.  Get the calorimeter crates. Verify with the Captain/DAQ shifter if
    you can take the Calorimeter crates out of the global configuration.
    The LED pulser runs should only be taken when there is no beam in
    the detector (no collisions, no halo, prior to shot setup and not
    during beam studies).

8.  Download the Trigger. In the Taker, click on ***Modify/Change
    Trigger*** and download file
    ***commissioning/cal/cal-icd-led-pulser-xxx***. Set ***Recording***
    ***on***.

9.  

10. Make sure that all low voltage and high voltage power supplies are
    on and working. In the Power Supply GUI, perform the following
    checks:

    a.  In the Preamp tab, verify the ICD low voltage power is on. The
        last row of cells which begins with ***ICD_LVCP_PW09*** should
        be green (except for those that are grey), and in the column
        titled ***STAT*** you should see ***0x3***. If not, first click
        on ***RESET***. If that does not work, click ***ON*** and if
        that fails, call the Calorimeter expert.

    b.  Verify the BLS power supplies in the North and South are on;

    c.  Verify that all of the preamp pulsers are off (Pulser 00 through
        12). Click on the ***PLS Mode*** tab, and make sure there are
        zeroes in every row and column, if not turn them off;

    d.  Click on the ***Mode*** tab, then click on all four orange
        buttons in the ***Global*** row: ***Global T&C Reset***,
        ***Global ADC Reset***, ***Global VBD Reset*** and ***Reset VBD
        CTRL***. Verify all cells in the ***BLS M*** column show
        ***Normal***, and the cells in the ***ADC M*** column show
        ***Unsuppressed***.

11. Check Trigger Bit 12 in the Pulser GUI. Check in the pulser GUI that
    that ***trigger bit 12 is on***: the colored boxes under the trigger
    bits should be set to all pink except for 12 which should be green.

12. Verify that the ICD high voltages are on. The ***HV Global
    Monitor*** GUI should be displaying ***light blue*** boxes, and the
    ***HVC Channel Monitor Display*** GUI should be showing ***green***
    boxes for the voltages and currents. Check page
    [193](#starting-the-calicd-hv-monitoring-gui) if you need more
    explanation of the HV GUIs. If you think there is a problem,
    carefully read the documentation for instruction.

13. Check the Voltage and Delay setting for the ICD LED pulser. Open the
    ICD LMB Interface:

> \>*setup d0online*
>
> *\>cd /home/d0icd/vme/*
>
> *\>./lmb_int.py &*
>
> *\>cd /online/examines/scripts*
>
> There are detailed instructions on how to use this GUI on the web at:
> <http://www-d0ol/www/groups/icd/controls/icd_lmb.html>. Please take a
> few minutes to read the documentation before proceeding further. For a
> simple "snapshot" of the ICD, use a voltage setting of ***8.6*** (V)
> and a delay setting of ***100*** (ns). Type these values into the
> ***ALL*** column and then click on the ***Download*** button. Click on
> the ***Read*** button to verify that the values you just entered
> appear in all four columns.

14. Start the Run. Return to the taker. Click on the ***Start*** button
    at the bottom. Enter your ***name***, type in the ***voltage*** and
    ***delay settings*** \[for example: ***7.8V, 10ns***\] in the
    comment field, toggle the ***Calibration*** option (not Physics, not
    Cosmics, not Test) and then click on ***OK***. The run should start
    within 15-20 seconds. At the bottom of the taker, you will see the
    ***current time***, the ***current run number***, the ***total
    number of events*** taken by this run, a ***rate*** in Hz and some
    number with a \"+\" sign in front of it which is just the number of
    events recorded by taker in the last 15 second cycle period, and the
    elapsed time of this run. Watch to make sure the ***rate increases
    to several hertz*** (depending on prescale). If you see no rate,
    there is a problem either with the calorimeter readout system or L3.
    Consult with the DAQ Shifter.

15. Run cal_elec for the ICD. Type ***start_cal cal_elec***. Once
    started, type ***init***, then ***icd***, then ***start***.

16. Display the Histograms. Return to the histoscope window. We
    recommend you make the histoscope window as tall as the screen and a
    little wider. If the event count is more than 16 in the xterm
    window, select ***File/Process*** then ***Connect to Process***, and
    a smaller window titled ***Available Processes*** will open with the
    item ***cal_elec ICD histograms (d0cal)*** selected. Click on
    ***Connect***. In the left hand side, you will see under ***Sub
    Categories***, ***CALELEC.HBOOK0***. Select with the mouse and click
    on the ***Open*** button at the bottom of HistoScope. A long list of
    histograms will appear in the right hand window according to
    Calorimeter crate and ADC card. You will notice that only
    calorimeter crates 0 (0x40), 1 (0x41), 4 (0x44), 5 (0x45), 6 (0x46),
    7 (0x47), 10 (0x4a) and 11 (0x4b) are listed, and only ADC cards 1,
    2, 4, 5, 7, 8, 10 and 11 are available. We are primarily interested
    in the ***Mean vs. ch*** and ***Sigma vs. ch*** histograms.

17. Print out the Histograms. Just in case the files don't get saved
    properly, we would like a printout of the histograms. Make a
    ***Multiple Histogram*** plot of the means for ***Crate 0 and Crate
    1***, ***Crate 4 and Crate 5***, ***Crate 6 and Crate 7***, and
    ***Crate 10 and Crate 11***, i.e. four pages with 16 histograms per
    page. Now ***Print*** these four pages and file them in the white
    binder called ***Calorimeter Reference Plots***.

18. Collect 7000 events. You will need to glance at the bottom of the
    taker now and then. If you see ***IPC_STATUS 0*** and ***Waiting for
    message***, the data taking has stopped. You can confirm this by
    looking at the bottom of the taker, and noticing a rate close to
    ***zero Hz*** followed by ***+0***. Consult with DAQ Shifter. We are
    interested in taking ***7000 recorded events*** for each run (for
    each combination of voltage and delay setting).

19. Stop the accumulating data and save the histogram files. If the
    number of events booked by histo in the cal_elec xterm window is
    greater than 200 events, return to the xterm window and type
    ***stop*** and then ***quit*** . If this does not work, ***CTRL-C***
    will stop the run. However, we would need to have the hbook file
    from each run so we can look at the mean and sigma histograms to get
    a quick idea of which runs to give highest priority for analysis. In
    ***/online/examines/output/*** there is a file that looks like
    ***cal_elec\_\_\$date\_\$time.hbook0***. If you can stop and quit
    normally, then go to that directory and rename (you can use
    ***./icd_rename_hbk 7.8 10*** to rename for 7.8V and 10 ns for
    example) the file like the following:
    ***21_1_2002_7.8V_10ns_ICD_LED.hbk*** (it should be clear what the
    filename format is, i.e. day_month_year_voltage_delay_ICD_LED). You
    can verify if the file was closed properly by looking at it within
    the HistoScope. Select ***File/Process*** then ***Open HBOOK
    File***, select the file of interest and click on the ***OK***
    button. If nothing shows up in the right hand side, and/or the
    ***Current Category*** is ***Uncategorized***, then the hbook file
    was ***not*** closed correctly. In that case please start over for
    this voltage and delay setting.

20. Stop the Taker and record run data. Click on the ***Stop*** button
    at the bottom of the taker. Type in the comment field the name of
    the ***hbook file***. Enter the following information into the
    logbook: ***Run number***, ***Voltage offset***, ***delay time***,
    ***Approximate number of events*** (see the bottom of the taker),
    ***Did a hbook file get saved***? Copy and paste this into an email
    and send to [mailto:d0icd@fnal.gov](mailto:d0icd@fnal.gov). There is
    a little whiteboard above the CAL/ICD console for Beam Off Tasks. In
    one space, I wrote ***ICD LED Runs Performed***. After you have done
    the run, write, for example: (***7.8V,10ns***) so the next shifter
    to take a ICD LED pulser run does not repeat that voltage and delay
    combination.

21. Run another if time permits. Each run will take approximately ten to
    fifteen minutes. If you have time, proceed to the next voltage and
    delay setting combination (see whiteboard for what has already been
    done). If there is not enough time, or the DAQ Shifter/Captain have
    requested the calorimeter crates be returned to the global run, then
    proceed to the next step and shutdown the pulser. This is ***VERY
    important***.

22. 

23. 

24. Free the Taker Trigger. Locate the taker, select ***Modify*** , then
    select ***Free Trigger***. Now you are done.

# BLS trip

1.  ***Undervoltage*** trip. Our experience tells us that a fuse in the
    BLS supply has melted. Try resetting the supply a number of times
    quickly. Sometimes that brings it up because it lets you charge up
    all the capacitances. If it does not reset, there is nothing to do
    but replace the supply and take out the old one for repair.

2.  ***Overcurrent*** trip. The supply is presumably ok, but there is a
    real load on the supply. In general, one of the BLS cards could have
    failed with a short on one of the voltage rails (e.g. a bad cap).
    You need to access the BLS crate and check whether a card has
    failed: pull out 8 boards out of 16, etc.

3.  A ***full quadrant*** (like for example ECS NW) of BLS supplies has
    tripped, i.e. all boxes in ***all six rows are pink***, and cannot
    be reset. This is an external interlock problem which you have to
    reset first. I n the appropriate BLS page (either ***BLS N***,
    ***BLS C***, or ***BLS S*** ), check if column ***STB8*** reads
    ***0x44***. If so, click on that particular box to get more detail,
    and you may see something like a box with ***Ext Int Fault*** in
    pink. The problem is likely a tripped rack monitor, so check the
    ***Rack Environment Monitor Display GUI*** (in Monitor #2). Select
    the ***Central***, ***North***, or ***South*** tab corresponding to
    the BLS that tripped (making the association is not obvious
    unfortunately, so look for any pink box!). Click the ***Reset***
    button for that row. A variant on this is that the Rack Environment
    Monitor Display looks ok (all ***green***) BUT it ***still needs a
    reset***. In that case try resetting them all (that way you are sure
    to get the right one).

4.  A ***full quadrant*** of BLS shows some ***pink/yellow***, in
    particular the ***+7VA I*** and the ***-3VB I*** columns (and
    sometimes others like ***-12VDI*** and ***+13VCI***), and all ***the
    rest*** ***is*** ***green***. In fact the currents are not off, but
    merely a little lower than normal (say 17A vs 19A). This problem can
    be caused when the ***clocks*** for the experiment (and the T&C in
    particular) go away. This sets the BLS in a lower power safe mode,
    hence the lower current draw. So, check to see that the ***CETEC
    supply*** (the last row) is on in the ***ADC tab*** in the ***Supply
    GUI***. If it has tripped, then clocks have indeed gone away, so try
    resetting it. If, however, the CETEC supply looks ok, then ask the
    DAQ shifter or the Captain if the experiment clocks are off. If so,
    then just wait until the clocks are restored, in which case the
    problem should go away once data starts flowing.

5.  ***A pair of BLS supplies*** (0 and 3 or 1 and 4 or 2 and 5) show
    ***pink*** ***STB8***, ***STCD***, and ***STEF*** boxes and a
    ***yellow*** ***STB4*** box. Everything else looks normal. It is
    just a readout problem and ***data taking is unaffected***. The
    ***STxx*** represent status bits that are read out from the BLS
    supplies, such as the state of the 3 phases, the temperature trips,
    etc. If they appear to have tripped (i.e. have turned pink or
    yellow), and yet all the currents and voltages look ok, then that
    means, most likely, that ***the digital cable has fallen out or come
    loose from the BLS supply*** in the platform. An authorized expert
    will request a short (15min) access to go in and reconnect the
    cable.

6.  ***A pair of BLS supplies*** (0 and 3 or 1 and 4 or 2 and 5)
    indicates that ***one (or more) of the 3 phases has tripped*** (i.e.
    turned pink). This means that one of the phases is removed due to
    the AC cord being twisted. An authorized expert has to "untwist" the
    AC cord during an access. If ***everything else looks OK***, the
    phase has tripped momentarily, but the warning light has latched the
    error, leaving the warning pink: ***the supply*** ***is*** ***fully
    working***.

7.  Old stuff (to be checked) *[Problem:]{.underline}* I pushed the top
    of a BLS supply and it tripped off. What happened?

> *[Solution:]{.underline}* The problem has been traced to the lugs on
> the transformer in the supply that are too close to the top cover
> (especially if they haven't been bent out of the way). They can short
> out the leads when the cover is depressed. We are working on adding an
> insulating layer on the cover to avoid this. In the meantime,
> releasing the pressure should solve the problem. However you should
> note in the e-log which supply this happened to.

8.  Old stuff (to be checked) *[Problem:]{.underline}* A BLS supply has
    tripped and it won't come back on. What do I do?

> *[Solution:]{.underline}*

a.  If it looks as if the voltages and currents are "trying" to come up,
    then it may be necessary to remove (temporarily) some of the load,
    i.e. pull out BLS cards from the crate, turn on the supply and see
    if it comes up (obviously that requires access). Then the cards are
    plugged in under power (carefully!).

b.  If it looks as if they are not "trying" to come up and you have an
    under voltage trip, then experience tells us this may mean that the
    internal fuse holder has melted. At that point the supply must be
    replaced with spares that are sitting in the collision hall.

Troubleshooting Guide

This chapter is exactly that -- a troubleshooting guide. Use it when you
run into problems or while waiting for the expert to call you back.

In this chapter we will try and compile a list of problems that have
arisen and their possible solutions. If you find a problem and can
explain how you fixed it, let us know and we will add it to this
section. It should grow to be quite useful.

  --------------------------- ------ ------------------------------ ------
  Software/GUI Systems        Page   Hardware Systems               Page

  Calorimeter Power Supply    76     Preamp Low Voltage Power       86
  Monitor Display GUI                Supplies                       

  Histoscope                  78     High Voltage                   90

  Data Taking Problems        80     Calorimeter Workstation        91

  Calib Problems              82                                    

  Alarm Server/Significant    83                                    
  Event Server (SES)                                                

  Downloading & Creating      84                                    
  Pedestals                                                         

                                                                    

                                                                    

                                                                    

                                                                    
  --------------------------- ------ ------------------------------ ------

# Software problems and failures

The list of categories for which we have documented troubleshooting
hints on software and GUI's are:

## Calorimeter Power Supply Monitor Display GUI

6.  *[Problem:]{.underline}* The GUI has stopped responding, all the
    boxes are greyed out, and clicking the ***Reconnect*** button
    doesn't do anything.

> *[Solution]{.underline}*: Exit the GUI by left-clicking on the
> ***Exit*** button on the lower right. Now restart the GUI by opening
> an xterm window and running ***start_cal supply***. If that doesn't
> work, check the next problem down below.

7.  *[Problem]{.underline}*: I cannot start the Calorimeter Power Supply
    Monitor Display GUI (= Supply GUI) from an xterm window. I keep
    getting the error message ***start_cal: Command not found***.

> *[Solution]{.underline}*: While the cause of this problem is not
> known, you can start the GUI up by hand. Run
> ***/online/config/cal/supplies/supply_all &*** from your xterm window.

8.  *[Problem]{.underline}*: A lot of preamp power supplies now read red
    -- in particular the currents for the secondary (unused) power
    supplies read something like -0.2 and the limits are -0.1. How do I
    make all that red go away?

> *[Solution]{.underline}*: This can be caused by having one of the
> front-end nodes (like d0olctlxx) reset. After a reset they get their
> limit data (these are the limits that set the yellow and red colors)
> from the database, and the database has not been updated so it has the
> incorrect limits. This is being worked on, but in the meantime you
> will need to contact Dave Huffman to set the correct limits by hand.

9.  *[Problem]{.underline}*: The Calorimeter Power Supply Monitor
    Display GUI (= Supply GUI) is behaving strangely. Some rows of boxes
    just show grey, and when I try to ***Reconnect*** (the box in the
    lower left) it occasionally fixes the problem row, but a new row
    gets the same problem! How do I fix this?

> *[Solution]{.underline}*: The problem can be that the computer
> resources are being used by some rogue process. You can check that by
> going to the ***IOC resource display*** GUI (probably on Monitor #3)
> and selecting the ***Cal*** tab and looking at the ***d0olctl03*** row
> and checking the ***CPU %*** and ***Mem %***. If either is above about
> ***70%***, you probably have a ***runaway GUI/process***. To find that
> process you will likely need to call a ***cal monitoring expert***
> (e.g. Leslie Groer).

10. *[Problem]{.underline}*: I just noticed that a large number of BLS
    current readings show pink, but then they return to normal green. Is
    that normal?

> *[Solution]{.underline}*: Possibly. If you notice that the current
> readings turn pink and data taking has stopped, then this is ok as
> long as they return to green once data taking resumes. For a more
> detailed explanation and solution see problem 4 on page 88). If it
> persists as pink after data taking starts, then there is a problem.
> Call the on-call expert.

## Histoscope

4.  *[Problem]{.underline}*: When you try to ***Connect*** ***to
    process...*** you get the following error message ***Error listing
    file: Error scanning /tmp directory***.

> *[Solution]{.underline}*: First make sure that you have collected at
> least 15 events -- otherwise you will get this error. If know that you
> have taken more than 15 events, then this workaround, which was not
> necessary in the past, seems to solve the problem. Hey, it's all black
> magic! You will need to type ***setenv TMPDIR /tmp/dummy_name*** (yes
> it means ***dummy_name*** exactly) in both the xterm windows from
> where you are running examine (or the cal_elec that is producing the
> histogram) and from where you are running histo. It is also possible
> that your problem is due to additional Examines that are running. In
> that case do a ***ps --u d0cal*** to identify those processes and then
> kill them.

5.  *[Problem]{.underline}*: I tried to start the cal examine process
    using start_cal examine, but it doesn't seem to run. Is there
    another way to get this process started?

> *[Solution]{.underline}*: The script is supposed to do everything for
> you, but sometimes there are problems because you might be on the
> wrong node etc. You might first try opening a fresh xterm window and
> trying again. If that fails you can try starting it by hand. That
> means
>
> \>setup d0online\
> \>d0ssh d0ol45\
> \>setup histo\
> \>histo &\
> \>setup d0online\
> \>cd /online/examines/scripts\
> \>./cal_examine.sh\
> *... after you see the message: command reader starting*\
> init\
> start
>
> *...open the histograms and look at plots, then finally...\
> ...when you are done...*\
> stop\
> quit

6.  *[Problem]{.underline}*: I tried to start the L1 cal examine process
    using start_cal l1examine, but it doesn't seem to run. Is there
    another way to get this process started?

> *[Solution]{.underline}*: The script is supposed to do everything for
> you, but sometimes there are problems because you might be on the
> wrong node etc. You might first try opening a fresh xterm window and
> trying again. If that fails you can try starting it by hand. That
> means

\>cd /online/examines/p12.04.00/l1calex

\>setup d0online

> \>setup D0RunII p12.04.00\
> \>d0setwa

\>cd l1cal_examine/online

\>./Runme.sh\
init

start

> *...and when you are done...*\
> stop\
> quit\
> *...in new xterm*

\>setup histo

> \>histo &

## Data Taking Problems

7.  *[Problem:]{.underline}* I have crates that are missing in the
    event. What do I do?

> *[Solution:]{.underline}* Go to the ***Mode Shift*** page (which you
> get to from the ***Mode Shift*** tab in the ***Calorimeter Power
> Supply Monitor Display GUI (= Supply GUI)*** ). Ask the DAQ shifter to
> ***pause the run***. Then click on the ***Global T&C reset***, then
> ***Global ADC Reset*** and ***Global T&C reset*** again. Ask the DAQ
> shifter to issue an ***SCL init***, and then ***restart the run***.

8.  *[Problem:]{.underline}* I see lots of ***0x30*** (pink) flashing in
    the 2^nd^ column on the mode shift tab in the Calorimeter Power
    Supply Monitor Display GUI (= Supply GUI) , but otherwise everything
    looks normal.

> *[Solution:]{.underline}* Don't worry it is normal. The ***0x30***
> simply means that it is waiting for the L2 decision.

9.  *[Problem:]{.underline}* Data taking has stopped, the DAQ shifter
    has told me one of the calorimeter crates is at fault, and I see the
    crate showing ***0x11*** in the 2^nd^ column of the ***Mode Shift
    tab*** in the Calorimeter Power Supply Monitor Display GUI (= Supply
    GUI) . What should I do?

> *[Solution:]{.underline}* The most prudent course of action is to ask
> the DAQ shifter to ***pause the run*** and issue an ***SCL init***.
> Issuing an SCL Init while we are running can cause all sorts of grief
> to the data. Do not do it yourself.

10. *[Problem:]{.underline}* The DAQ is looking to me to reset the Cal
    crates because of a problem. What should I do?

> *[Solution:]{.underline}* Don't believe him/her. The first thing you
> should do is ask the DAQ shifter to ***pause the run*** and ***issue
> an SCL Init***. If that solves the problem you are done. If not, then
> you can ask the DAQ shifter to ***pause the run*** again, and you
> issue an ***T&C reset***, then an ***ADC reset***, then a ***T&C
> reset***. Make sure that after that the DAQ shifter issues a ***SCL
> Init***. If that hasn't solved your problem, it is time to call the
> on-call expert.

11. *[Problem:]{.underline}* My calorimeter console node seems to have
    slowed down a lot. What can I do?

> *[Solution:]{.underline}* You might have a number of "rogue" processes
> running (such as a bunch of Examines). You need to find them and kill
> them by doing \[here this is a placeholder for instructions that I
> should get from Alan Stone\]. In the meantime, try looking at page 91
> which tells you how to find the processes even if your console appears
> frozen.

## Calib Problems

4.  *[Problem]{.underline}*: There appears to be something wrong with
    the Calib GUI and I see many zeroes in the calib printout. Any
    ideas?

> *[Solution]{.underline}*: If you ***exited improperly*** from the
> Calib GUI. For example you did an ***Exit*** from the GUI when the
> ***ABORT*** button was showing on the bottom left, then you may have
> created detached processes that corrupt your calib data. You should
> first try restarting the Calib GUI and exiting properly (i.e. click
> the ***ABORT*** button, or wait until ***CONFIGURE*** appears in the
> GUI and then Exit). If that fails you should call the on-call expert,
> who in turn may call the Calib expert (Taka).

5.  *[Problem]{.underline}*: The calibration process looks like it never
    completed. What can I do?

> *[Solution:]{.underline}* This is caused when a previous calibration
> run did not completed or it was not aborted properly. This can happen
> when a second calibration run is started before the first one is
> completed or when a calibration run is stopped from a taker window
> instead of from a Calib_Manager gui window. I have put in a fix to
> deal with this situation. But it still happens once in a while due to
> subtle timing in which itc processes communicate. So, the current fix
> is to kill ***processes that are left over from the previous
> calibration runs***. To do that, logon to ***d0ol08*** as ***d0run***
> and type ***ls -f -u d0run***. At the end of this paragraph we have
> included the text you would see when doing this for a specific example
> with the database interface process called Subsystem_Validator.py.
> Type ***kill -9 pid*** (where ***pid*** is the process ID) on the main
> thread. The threads created by the main thread will die when the main
> one is killed. In the example , the ***pid is 5810***. You might have
> to kill more than one such processes, if somebody tried to run the
> calibration  more than once after Calibration Manager got into this
> state. After killing all these processes, you should be able to take
> calibration runs again. Do not kill Calib_Manager for this.  Problems?
> Contact Taka.

 

> d0run 5810    1 5 09:54 pts/9 00:00:01 python
> /online/products/onl_calib_system/onl00-03-00/Linux+2/py/Subsystem_Validator.py
>
> d0run 5811 5810 0 09:54 pts/9 00:00:00 python
> /online/products/onl_calib_system/onl00-03-00/Linux+2/py/Subsystem_Validator.py
>
> d0run 5812 5811 0 09:54 pts/9 00:00:00 python
> /online/products/onl_calib_system/onl00-03-00/Linux+2/py/Subsystem_Validator.py

6.  *[Problem:]{.underline}* The peds.txt link has the wrong date. What
    do I do?

> *[Solution:]{.underline}* You need to establish the proper link by
> hand. If you really don't know what you are doing, call the on-call
> expert.\
> *\> ln --sf valid/peds_ped_yymmdd_hh:mm:ss.txt peds.txt* (short 500evt
> run)
>
> **OR**
>
> *\> ln --sf valid/pref_ped_yymmdd_hh:mm:ss.txt peds.txt* (long 10kevt
> run)

## Alarm Server/Significant Event Server (SES)

2.  *[Problem]{.underline}*: The alarm server GUI has a major alarm
    showing (a pink box). How do I find out what has gone wrong?

> *[Solution]{.underline}*: First you should left-click on the pink box
> in the alarm server GUI. That will pop-up a more detailed window with
> information about what specific device caused the error. The names can
> be cryptic, but the name should be somewhat informative -- for example
> you may see the same name in the Calorimeter Power Supply Monitor
> Display GUI (= Supply GUI) . You will probably also see a pink box in
> the power supply monitor GUI (again left-click on the box for more
> details) which should help you to localize the problem to a specific
> power supply. Once you have identified the problem, reset the
> appropriate supply (if that is the problem), and note it in the
> e-logbook. If you can't find or fix the problem, call the on-call
> expert.

## Downloading & Creating Pedestals

4.  *[Problem:]{.underline}* I completed taking the ***x1*** and
    ***x8*** pedestal runs but when I ran the ***make_peds.sh*** script
    I got the following error message:

> \....
>
> \.....
>
> set run8=160564
>
> set run1=160560
>
> ./cal_peds_query_x8_x1.py \$run8 \$run1
>
> make_peds takes about 2 minutes per crate
>
> crate 64 is being done
>
> Traceback (most recent call last):
>
> File \"./cal_peds_query_x8_x1.py\", line 236, in ?
>
> main ()
>
> File \"./cal_peds_query_x8_x1.py\", line 229, in main
>
> writePickle(name3, dataList)
>
> File \"./cal_peds_query_x8_x1.py\", line 67, in writePickle
>
> crateClass.readList(dataList, dataDelim, killValue)
>
> File \"./pickleClass.py\", line 93, in readList
>
> self.dataTable\[adcCount\]\[dataOffset\] = int(listname\[listOffset\])
> \#
>
> Load gain x8 pedestal value into dataTable
>
> IndexError: list index out of range
>
> \....
>
> \....
>
> What is wrong, and what do I do?
>
> *[Solution]{.underline}*: The problem is that one of the runs was not
> fully committed into the Database. Verify that there is indeed a
> problem by typing in an xterm window ***setup oracle_client***, then
> ***sqlplus cal_calib/cal_calib@d0onprd1*** now in the program ***type
> sql\>select count (\*) from pedestals where run_number=xxxx***, where
> ***xxxx*** is the run you would like to check. If the answer is
> anything but ***55296*** (the number of cal channels), then there is
> indeed a problem. In that case, if you have time, take the pedestal
> calibration runs again. If you don't have time, call the on-call
> expert.

5.  *[Problem:]{.underline}* I tried to download pedestals using Taker's
    ***cal_prepare_for_run-xxx (where xxx is the version number and if
    you find more than one version call the on-call expert)*** trigger.
    All the crates failed to download with a ***devdnl timeout*** error.

> *[Solution:]{.underline}* First just try it again. If the problem
> persists, then this problem has been seen to be associated with a
> comics download problem. Ask the DAQ shifter to stop comics
> (***stop_daq comics***) and restart it (***start_daq comics***).

6.  *[Problem:]{.underline}* I tried downloading the pedestals using
    ***cal_prepare_for_run-xxx (where xxx is the version number and if
    you find more than one version call the on-call expert)*** and
    although most of the crates appeared to download correctly, a few
    failed. What do I do?

> *[Solution:]{.underline}* While we are unsure of the cause of the
> failure, the best advice we can give at the moment is to ***try the
> download again***. If that fails, it may be that you have ***too much
> CPU useage on d0clt03***. To see if that is the case, look at the
> ***IOC Resource Monitor Display GUI*** (it should be on the monitor
> above your head), click on the ***CAL tab*** and check the ***CPU%***
> usage for ***d0ctl03*** -- if it is large, ***close all power supply
> GUI's*** and if that still does not get you below 40%, then ***close
> the CRATE MONITORING GUI (=T&C GUI)***. If it really keeps on failing,
> call the on-call expert. If you closed GUI's, then make sure to open
> them again once the problem is resolved.

## Channel Archiver

3.  *[Problem:]{.underline}* The Archiver cell in the lower left has
    turned pink. What does that mean?

> *[Solution:]{.underline}* The channels archiver that records the state
> of all currents, voltages, etc has stopped running. See below on how
> to start it up again.

4.  *[Problem]{.underline}*: The channel Archiver has stopped running.
    How do I start it up again?

> *[Solution]{.underline}*: The archiver can be pretty finicky. Dave
> Huffman is our local expert, and Vladimir Sirotenko is the D0 expert.
> For detailed instructions check
> <http://www-d0online.fnal.gov/www/groups/cal/Archiver.pdf>, but here
> is a quick summary:\
> *\> setup d0online\
> \> setup chan_archiver\
> \> cd /projects/archive/cal/current\
> \> rm archive_active.lck\
> \> start_chan_archiver --c config_file --p 4814 --w . --d
> dir.xxxxx-yyyyy* (where dir.xxxxx-yyyyy is the directory file name
> that has stopped, e.g. dir.20011019-091652)

# Hardware Problems and Failures

The list of categories for which we have documented troubleshooting
hints on hardware problems and failures are:

## Preamp Low Voltage Power Supplies

2.  *[Problem:]{.underline}* The preamp supply has tripped and will not
    reset. What do I do?

> *[Solution:]{.underline}* Because of the difficulty of accessing
> preamp power supplies during running, they have been made redundant.
> That means you can turn off the "bad" supply and turn on the secondary
> supply. When you look at the Calorimeter Power Supply Monitor Display
> GUI (= Supply GUI) (if it is not going, then run ***start_cal
> supply*** in an xterm window) and select the preamp tab (on top), you
> will notice that the supplies are labeled something like
> ***CALC_LVCP_PA02P*** and right under it is another labeled
> ***CALC_LVCP_PA02S***. The one that ends in "***P***" is the primary
> supply and the one that ends in "***S***" is the secondary supply. The
> way you can tell which one is being used is by noting the currents --
> the active supply will show non-zero currents, whereas the spare
> supply will show very small or zero currents. Use the buttons on the
> right hand side of the GUI to ***turn off the bad supply*** and
> ***turn on the spare***. Obviously you should leave a note for the
> experts telling them about this. It has recently been noted that when
> one supply tripped BOTH primary and secondary showed a red box. In
> that case, the tripped supply was turned off and the other was turned
> on, then both became green again -- try that procedure first before
> calling an expert.

## High Voltage

4.  *[Problem]{.underline}*: I see a major alarm for the HV. What do I
    do?

> *[Solution]{.underline}*: This normally means that the HV has tripped
> (this has occurred on occasion during shot setup). You can check that
> on the HV display (the one that normally shows a lot on blue boxes).
> If there is a trip, you will see one (or more) of the boxes in red. If
> it is an ICD HV trip, then follow the instructions in the HV section
> (p. 194, because you will need to turn off the ICD preamps. Otherwise
> if it is a calorimeter HV trip, then left click on that GUI in the
> section that has the trip -- that will pop-up another GUI with the
> details of the HV (setting, readings, status etc), the offending HV
> channel will be pink. ***Left click on the pink box*** in this GUI and
> select ***unlock***. Now left click again and select ***ramp***. After
> some time the voltage should ramp back up to its set point. Click on
> the ***Lock*** button on the bottom of the GUI and you will see the
> boxes ***turn turquoise and say locked***. That's it.

5.  *[Problem]{.underline}*: I tried to reset the HV, but it doesn't
    ramp up. How do I get the voltage up?

> *[Solution]{.underline}*: Occasionally the voltage won't ramp up all
> at one time. In that case try ramping up the voltage in steps -- go to
> 10%, then 20%, etc until it gets up to 100%. In order to set the
> voltage sets, look in the HV GUI for the ***Set HV*** tab, it will
> show various percentages, ***check the box*** (indicated by a red
> diamond) for the appropriate voltage percentage, and then ramp it up.

6.  *[Problem]{.underline}*: For some reason one of the ***HV has been
    set to 0***. When I click in the ***V_Set*** box, it turns yellow
    and I appear to be able to change the setting. But when I click out
    of that box the setting goes back to zero. How do I set the voltage?

> *[Solution]{.underline}*: The way you can select the appropriate
> setting for ***V_Set*** is to look in the HV GUI for the ***Set HV***
> tab, it will show various percentages, ***check the box*** (indicated
> by a red diamond) for the appropriate voltage percentage, and then
> ramp it up. So if you want the full ***V_Max*** voltage (which is
> usual), then select ***100%***.

## Calorimeter Workstation 

2.  *[Problem:]{.underline}* I cannot move the mouse and have lost total
    control of my workstation. What do I do?

> *[Solution:]{.underline}* There are a number of things you can do.
> First try and identify the process which is causing the problem (for
> example too many Examines?), then type ***CRTL-ALT-F1*** or ***F2***
> (all screens will go blank and you will be prompted to login). Login
> as ***d0cal***. Type ***ps --ef \| grep d0cal***, then type ***kill -9
> process#*** where process# is the exactly that, the process number you
> see listed. Now return to the normal screen by typing
> ***CTRL-ALT-F7***. You can repeat this procedure as needed to kill
> other jobs. If there many Examines that you need to kill, then you can
> kill them all at once by typing ***killall CalExamine_x*** or
> ***killall L1CalOnlineExamine_x*** for example -- be careful because
> killall is a powerfull command! If that fails, then I'm afraid you
> should ***reboot***.

[]{#__RefHeading___Toc59549922 .anchor}Power Outage

# Overview

Normally (as if anything is ever normal) you should never have to deal
with recovering from either a planned or unplanned power outage. Please
contact an expert to turn on after a power outage, but if you are on the
owl shift and no-one else is around you may need these instructions.
Besides turning on the computers and power supplies, you will need to
reload the pedestals in the ADCs.

# Preparing for a power outage

1.  Turn down Low Votage Pulser.

> In the Power Supply GUI, Press \"Turn Off All\" and confirm in the
> following

tabs: Preamp (including ICD), BLS N, BLS C, BLS S, ADC and Pulsers.

2.  Turn down High Votage Pulser.

In the HV GUI, click on an ICD blue horizontal bar and on a CAL blue
bar. This opens 2 HVC Channel Monitor displays. These displays have
tabs: NE, NW, SE, SW for ICD and N, C, S and Argon Mon for CAL. The
following has to be done on each of these tabs:

a.  press \"Unlock\" at the bottom right of the screen. States turns
    from locked to Holding.

b.  Select 0% in the Set HV Menu at the top left of the page. Column
    V_Set turns to 0, V_Read becomes pink (V_Read differs from V_Set)
    and the Ramp button becomes pink.

c.  Press \"Ramp\" at the bottom of the page. The voltages ramp down to
    zero. That may take some time. Some individual channels may pause.
    You left click on the state and select \"Resume\" (or do it globally
    from the bottom). When the voltages are below 100V, you have to turn
    them \"Off\" either individually by left clicking\"Off\" in the
    State box.

```{=html}
<!-- -->
```
3.  Turn down the supplies themselves.

    a.  BLS low voltage supplies: in the 36 BLS power supplies located
        below the detector, throw the white breakers. Put the 3 little
        switches on \"off\", \"local\", \"off\" to prepare for the
        restart. At the same time, power off the

    b.  Pulsers low voltage supplies: the pulser power supplies are
        located below the BLS power supplies. Throw the breakers and
        switch the supply to \"local\"

    c.  ICD low voltage supplies: in rack PW09, on the west side of the
        detector, throw the breaker at the bottom left of the rack.
        Throw also the pulser breaker and switch the supply to
        \"local\".

    d.  Preamplifiers: throw the 30 breakers (24 black + 6 white) in
        PW06 (rack facing PW09).

    e.  ADCs: In MCH3, throw the breakers of the 6 power supplies (3 in
        MC307 and 3 in MC311) and put the 3 little switches on \"off\",
        \"local\", \"off\" to prepare for the restart.

    f.  CETEC: put the 2 little switches to \"local\", \"off\" (in crate
        MC307)

    g.  unplug the cables from the back of the 2 computer crates: MCH300
        bottom crate and MCH309 crate at eye height.

    h.  turn off HV low voltage supplies in M117 (7 white breakers; you
        can also switch off the 5 green fans and the LEVEL0 supply).

4.  5K test stand.

    a.  Throw the breakers of the BLS and pulser supplies.

    b.  Upstairs, switch off the ADC power supply and throw all breakers
        in the power distribution chassis

# Restarting Computers & GUIs

We will assume that all "other" computers have been turned on by others
like the shift captain. But just in case, you may need to turn on
d0ol45. That computer is attached to the three screens that are located
at the CAL/ICD console in the D0 control room. The CPU itself can be
found behind the Monitor #1. You may also need to reboot the d0ol32
computer, which is attached to the 4 displays that sit on top of the
CAL/ICD, CONTROLS, CFT/CPS/FPS, and MUON consoles. The CPU is located at
knee level in the CONTROLS console -- open the red door to access it.

1.  Reboot d0ol45 and d0ol32. To power-up these machines, just press the
    pushbutton.

2.  Log into D0OL45.

3.  Open a new xterm window.

4.  Start the GUI's

# Recovering from a Power Outage

Undo whatever was done for the power outage preparation.

1.  Restart 2 EPICS nodes. Plug the cables in the back of the 2
    computers in MCH300 bottom crate. Don\'t plug in computer in MCH309
    crate yet.

2.  In the Supply GUI. In the BLS N page, press Turn All Off and Reset.
    Do the same for BLS C, BLS S, ADC and Pulser pages.

3.  Turn on the supplies themselves.

    -   BLS low voltage supplies: in the 36 BLS power supplies located
        below the detector, make sure the 3 little switches on \"off\",
        \"local\", \"off\". Throw the white breakers on and press the
        reset button. If the crate does not reset (some red LED except
        the ON/OFF one), hit the reset button several times. If it does
        not reset, call an expert. Put the central switch to \"remote\";
        don\'t worry if some crates will power on while other won\'t.
        Make sure the RMI have 4 green LED lit. If one is not, press the
        Reset button. If it does not reset to green, report to an
        expert. At the same time, power on the

    -   Pulsers low voltage supplies: the pulser power supplies are
        located below the BLS power supplies. Throw the breakers on and
        switch the supply back to \"remote\".

    -   ICD low voltage supplies: in rack PW09, on the west side of the
        detector, throw the breaker at the bottom left of the rack on.
        Put it on remote (second button \"G=Remote\"). Throw also the
        pulser breaker on and switch the supply to \"remote\".

    -   Preamplifiers: throw the 30 breakers (24 black + 6 white) in
        PW06 (rack facing PW09).

    -   ADCs: In MCH3, throw the breakers of the first power supply and
        put the 3 little switches on \"off\", \"remote\", \"off\". Hit
        the reset button and **wait until the SBC beeps** (not to crash
        the boot server). Do this for the other 5 supplies. Make sure
        the RMI have 4 green LED lit. If one is not, press the Reset
        button. If it does not reset to green, report to an expert.

    -   Don\'t turn on the CETEC crate yet!!!!!

4.  Now go to the \"Restarting Power Supplies\" section.

# Restarting Power Supplies

1.  Reconnecting the GUI's. If any GUI is opened, exit from it (Hit Exit
    in the File tab). Restart all the GUIs by typing \"start_cal all\".
    If the GUIs have no connection, most probably the EPICS nodes booted
    before the host was ready to serve the information. You have to
    reboot these nodes by pushing the RST button on the 2 CPUs
    (d0olctl09 and 11) at the bottom of rack M300. Then you will have to
    hit the Reconnect button on every GUI.

2.  Rack Monitoring Interfaces (RMI). Find the 2 Rack Environment
    Monitor Display GUIs on the upper left screen. If you don\'t find
    them, type \"start_cal rmi\" in a window of the upper left screen.
    One is for the platform (pages for Central, North, \...). Only pages
    Central, North, West, South and Cath(edral) are calorimeter related.
    In North, South and Cathedral, all lines are calorimeter. In
    Central, all lines except PC03/04/19/20 are calorimeter. In West,
    all lines except PW03 are calorimeter. The other is for the Movable
    Counting House (MCH). Only pages MCH1N and MCH3S are relevant for
    the calorimeter. In MCH1N, only line M16/17/18 are calorimeter. In
    MCH3S, all lines except M301/02/03 are calorimeter. All relevant
    (see above) RM DSTAT should be normal and no box should be pink. If
    not, click the reset button. If that does not do anything, may be an
    expert can help.

3.  Preamp Supplies. In the Calorimeter Power Supply Monitor Display
    GUI, select the \"Preamp\" tab and click on Turn OFF all (confirm),
    next \"Reset All\". Column STAT on the right should all display 0x2
    (Power Off). If a box is pink, click the Reset button on the same
    horizontal line. If that does not clear it, record the fault (left
    click on the pink box) and contact an expert (who will call Tom
    Regan). If the pink box shows 0xffff, no AC power is getting to the
    power supply. May be someone turned off the breaker intentionally,
    may be the power supply failed. In both cases, a failed primary
    (resp. secondary) supply makes its fellow secondary (resp. primary)
    supply have voltage boxes be pink as well. Now click on either the
    \"Turn On Primary\" button or the \"Turn On Secondary\" button
    (lower left). You must ask experts which is available because we
    periodically exercise the primary and secondary supplies. Of course
    if there is an individual supply that fails to come on, then you
    must try the other one. If a power supply failed, record its name
    along with the voltage and/or current. You should also check that
    only primaries or only secondaries are on. If not, turn them on a
    second time.

4.  BLS Supplies. In the Calorimeter Power Supply Monitor Display GUI,
    select the \"BLS N\" tab and click on \"Turn OFF all\" (confirm),
    next on \"Reset All\". If \"STB8\" is not 0x40, record the fault
    (left click on the pink box) and contact an expert (who will call
    Tom Regan). Columns STAB, STCD and STEF should be 0x0. If not, try
    individual resets and if still not, record the fault (left click on
    the pink box) and contact an expert (who will call Tom Regan). Then,
    click on the Turn ON all. You might have to Reset individual
    supplies (multiple clicks may be needed). Also, +7VA and -3VB
    currents may be pink. Don\'t worry at that moment. Now select the
    BLS C and BLS S tabs and turn those on in the same way.

5.  ADC Supplies.

    a.  In the Calorimeter Power Supply Monitor Display GUI, select the
        \"ADC\" tab and click on \"Turn OFF all\" (confirm), next on
        \"Reset All\" and on "Turn ON all".

    b.  Walk to MCH3 rack MCH309 crate, unplug (if not already
        unplugged) the power chord of the EPICS node and replug it.
        Return to the control room.

    c.  Type start_cal vme_adc_north and fill the left column with 41
        (hit return!) 6 times and quit. Do the same for south as you did
        for north. This sets the ADC crate controllers in the special
        mode \"readout disable=0x41\".

    d.  We must now power up the CETEC power supply (T&C crate supply)
        which is not yet computer controlled (volunteers?). Walk to MCH3
        rack M307, turn the little switches to \"local\" and \"on\" on
        the T&C power supply (labelled CETEC) near the top of the rack.
        Reset the VIC (Vertical InterConnect) in the T&C crate. Wait 15
        to 20 seconds (you could hear a beep) then reset (plug/unplug)
        EPICS node in rack M309. Wait about 1 minute and observe if any
        BERR LED shows red in the 4 VIC masters. Return to the control
        room.

    e.  You should be reconnected in Mode and Mode Shift pages. The FPGA
        column in the Mode tab of the Supply GUI should be the current
        FPGA version (32 on October 19, 2003). If not, reset (the
        button!) EPICS node and try again downstairs.

    f.  You are ready to program the T&C in the T&C GUI (if not found,
        type \"start_cal tandc\"). Click on Global T&C Set purple
        button, select Normal, go to CELL and type 0 and Enter (0x0
        should appear); go to the TICK box and do the same. Close the
        box. Click T&C CTRL on the botom of the page. Enter MODE 8089
        and press Enter (it sould echo 0x8089 in blue) and DIAG f0 (not
        0xf0) and press Enter (it sould echo 0xf0 in blue). The T&C mode
        column in the \"Mode Shift \" tab should be green; if not, call
        an expert.

    g.  Back in the T&C GUI, click now on the Global BLS Mode and select
        Normal (close the box). Click now on the Global ADC Mode and
        select Sel0SignSup (does not matter really). In the Mode Shift
        page, ignore ADC ERR, PED VERS, PULSER and OCC until the first
        event is taken. Everything else should be green, especially
        STATUS should be 0x10. If not, reset individual T&C.

6.  Pulser Supplies. In the Calorimeter Power Supply Monitor Display
    GUI, select the \"Pulser\" tab and click on \"Turn OFF all\"
    (confirm), next on all 13 individual \"Reset\" buttons (presently
    \"Reset All\" does not seem to work). Column STAT on the right
    should all display 0x2 (Power Off). Finally, hit \"Turn ON All\". If
    necessary, \"Shutdown\" the pulsers from the Set Calorimeter Pulser
    GUI (purple bottom left button); may be this will be done by
    downloading the file from a taker.

7.  HV supplies. Go to 193.

## 

## 

## 

## 

## 

## 

.

# If the ADC crates have been Power Cycled

If the power to the ADC crates has been cycled, then you will need to
reload the pedestals. You should do this anyway after extensive work on
the calorimeter system, which includes power cycling.

To download the pedestals and set the correct data taking state for the
electronics, follow these steps -- but first make ***sure to ask the
shift Captain for permission*** to download pedestals and inform the DAQ
shifter (since it can interfere with all other runs for the 5-10min it
takes). Make sure the calorimeter crates are not used in any run -- if
so ask the DAQ shifter to stop those runs and release the calorimeter
crates for your use. If a ***taker*** is not running somewhere, start
one by running ***setup d0online*** and then ***taker &***. In the
Taker, click on ***modify***, and then click on ***Change Trigger***.
You will now see the configuration window. To make sure you are in the
correct directory, click ***UP (Left)*** until you are all the way "up"
and the directories no longer change. Double click on the directory
***commissioning***, then double click on ***cal***, then double click
on ***cal_prepare_for_run-xxx (where xxx is the version number and if
you find more than one version call the on-call expert)***. It will tell
you "download in progress". This will among other things download the
pedestals. When you are finished, make sure to free the trigger. You do
that by clicking ***modify*** in the Taker, and then selecting ***Free
trigger***.

The above steps should have set the electronics in their correct state,
but just in case they didn't, you could also set the ADC and BLS modes.
Set the ADC mode to ***0 Sign Sup*** and the BLS mode to ***Normal*** --
at least that is a good "normal" starting value. After you have set
those values, then click on ***Global T&C Reset***, then ***Global ADC
Reset*** and finally ***Global T&C Reset*** again. That should do it.
This will occur automatically when you start running following the usual
run start procedures, but it can't hurt.

# If the T&C Crates have been Power Cycled

It is probably wise to set the T&C system after a power cycle. The steps
are similar to those you use to set the mode.

This method is in flux, but follow these instructions for the moment.
However, it may already be set properly, which you can check by looking
at the ***T&C CTRL*** row in the ***Mode*** tab of the ***Calorimeter
Power Supply Monitor Display GUI (= Supply GUI)*** -- if the ***brown
Mode column*** (the 1^st^ column) is set to ***0x89***, and the 4^th^
column is set to ***0xf0***, then you are done. Otherwise, in an xterm
window change the directory, ***cd /online/config/cal/vme***, execute
the usual ***setup d0online***, and then run ***./vme_tc_ctrl &***. Now
select (left click and hold while you sweep over the entry with the
mouse cursor) the number in the 1^st^ column and type in ***89***.
Similarly for the 4^th^ column, type in ***f0***. These values should
now be in the boxes (with a 0x in front of them). You are done, so
***Quit***.

[]{#__RefHeading___Toc59549930 .anchor}Detector Performance Monitoring
Software

# Overview

The programs that can be used to monitor the state of the electronics
and the data during data taking (and for debugging and commissioning)
are called cal_Elec and cal_Examine. Normally you would not recompile
these programs, but if you have to, then you should log into d0ol02.

# Cal_Elec

## Overview

This program is used to check the calorimeter electronics, both for
debugging and commissioning and during data taking. If you need to
recompile ***cal_elec***, only ***setup D0RunII t01.xx.xx***. Otherwise
you might get a KAI error message.

## Running the software for debugging

In order to run the software you need to

1.  Change directory and setup. You should ***cd
    /online/examines/scripts***. Then run ***setup d0online*** and
    ***p13.06.00*** and finally ***d0setwa***.

2.  Edit the rcp file. You should edit the rcp file to place the program
    in the correct mode. Use you favorite editor or follow these
    directions to start a WYSIWYG editor if Unix is not your friend.
    Edit the file by typing ***xemacs cal_elec_rcp/cal_elec_HS.rcp***
    and set ***PEDS=true*** and all others ***false***. Then save and
    exit the editor. This should set you up with the "normal"
    conditions. Experts may want to set it in other modes.

3.  Start histo. You can start histo by typing ***histo &*.** This will
    pop up a window where you will select the histograms you wish to
    look at (see below). Next change directory by typing cd
    ***/online/examines/scripts/*** and then ***setup d0online***.

4.  Run cal_elec. Run the following ***./cal_elec.sh*** and type the
    following (you can go ahead and type ahead before the prompts show
    up if you wish):

> *init\
> 1* (this is the number of crates so you choose this number, max of 12)
>
> *0* (this is the crate you want, if you are doing two crates then
> enter the first crate number, carriage return, and then the second
> crate number)\
> *start*
>
> This could take some time -- be patient! When you are done, you can
> type\
> *stop\
> quit*

5.  Open the histograms. To do that, return to the histo pop-up box.
    Select ***Connect to Process***, then select the histograms you want
    to look at (highlight them with the mouse), and finally select
    ***View Multiple*** to display the histograms.

6.  Pop up the Cal Preamp map/conversion GUI. It can be very helpful
    while debugging to have the GUI that allows you to translate the ADC
    channel number to BLS/Tower/Depth. To run this GUI change directory
    ***cd /online/config/cal/map*** and run ***./cpmap &***. Just type
    in the values in any box, and the others will change to reflect the
    translation of that channel in all possible types of coordinates.

## Running the software to monitor a ramp

When using the pulser system to generate signals that ramp up (or down)
the calibration pulse, you will need to monitor the ramp. These
instructions tell you how to do that.

1.  Change directory and setup. You should ***cd
    /online/examines/scripts***. Then run ***setup d0online*** and
    ***setup D0RunII t02.08.00*** and finally ***d0setwa***.

2.  Start histo. You can start histo by typing ***histo &*.** This will
    pop up a window where you will select the histograms you wish to
    look at (see below).

3.  Run cal_elec_ramp. Run the following ***./cal_elec_ramp.sh*** and
    type the following (you can go ahead and type ahead before the
    prompts show up if you wish):

> *init\
> expert*
>
> *0* (this is the crate you want, choose from 0-\>11)
>
> *0* (this is the ADC card you want, choose from 0-\>11)
>
> *12* (this is the number of channels you want)\
> *start*
>
> This could take some time -- be patient! When you are done, you can
> type\
> *stop\
> quit*

4.  Open the histograms. To do that, return to the histo pop-up box.
    Note that the pulser ramp parameters should also have been
    downloaded in other instructions on the pulser and linearity ramps
    (but are not specific to cal_elec_ramp). Select ***Connect to
    Process***, then select the histograms you want to look at
    (highlight them with the mouse), and finally select ***View
    Multiple*** to display the histograms.

## Running the software for monitoring

1.  Change directory and setup. You should ***cd
    /home/d0cal/RUNEXAMS/cal_elec_test/cal_elec/rcp***. Then run
    ***setup d0online*** and ***d0setwa***.

2.  Start histo. You can start histo by typing ***histo &*.** This will
    pop up a window where you will select the histograms you wish to
    look at (see below).

3.  Run Cal_Elec. Run the following ***./run_NET_HS_EXPERT.sh crate1
    \[crate2\].*** Where the values for crate1/crate2 can take on any
    value between 0 and 11 (remember that we are limited to two crates
    at a time).

4.  Pop up the Cal Preamp map/conversion GUI. It can be very helpful
    while debugging to have the GUI that allows you to translate the ADC
    channel number to BLS/Tower/Depth. To run this GUI ***cd
    /online/config/cal/map*** and run ***./cpmap &***. Just type in the
    values in any box, and the others will change to reflect the
    translation of that channel in all possible types of coordinates.

## Error messages

# Cal_Examine

## Overview

The Examine software is designed to allow you to monitor the health of
the detector by looking at a variety of plots that show things like
detector cell occupancies, energy deposits vs eta and phi, and so on.
There will be reference plots available for the shifter to compare the
running data to. We are now running the Root version as the default, but
there is a Histoscope version. Instructions for running both versions
are shown below. Root is now in RCP so this is the display that should
be used, histo probably won't work now.

## Running the Root version of the software

1.  Open a new xterm window. Click the left-button on the mouse in any
    open space on your window, then select ***New Shell*** to open an
    xterm window. It is prudent to run ***setup d0online*** once it is
    open -- it sets up what you will need, and it never hurts. Examines
    should be run on d0ol23 node which is faster and relieves the load
    from d0ol45. If there is an Examine GUI open, then you are done. If
    not, the open a new xterm window (on the left screen usually), type
    the usual ***setup d0online*** and then ***d0ssh d0ol23***.

```{=html}
<!-- -->
```
1.  Check to see no other Examines are running. Now check to see if any
    examines or xgooey processes are running by typing ***ps --ef \|
    grep Xgooey*** and then ***ps --ef \| grep examine***. Kill those
    processes using ***kill -9 xxx*** where xxx is the process number.

2.  Start the program. In the xterm window ***setup d0online*** and run
    ***start_cal examine***. This opens the ***Examine Control Panel***.
    Click on the ***Edit*** button and select ***Preferences***. Change
    ***EXAMINE*** to ***d0ol23.fnal.gov*** and check the ***Canvas Menu
    Bar*** option under ***DISPLAY***. Back on the main control panel,
    click the red ***Start*** button (upper left), which will bring up a
    list of histograms.

3.  Choose one or a set of histograms that you would like to monitor.
    Select them with a left mouse click (at the moment ***do not select
    Total Energy, Total Et, EM energy, Had Energy, EM cell energy
    ieta/iphi, HAD cell energy ieta/iphi***).. Now click on the
    light-green ***Display*** button in the ***Histograms*** menu (which
    is at the bottom of the ***Examine Control Panel***.). That will pop
    open a window with those histograms. Don't forget to unselect the
    histograms before opening a new display (or "canvas" in root
    language), otherwise you will see them again! You can play with
    adjusting sizes, scales etc. Keep the number of histograms displayed
    to as small a number as possible. The system tends to slow down or
    hang otherwise. You can report problems in an "observation file"
    which you can copy (with xemacs) into the logbook.

## Running the Histoscope version of the software

This probably does not work anymore. But I leave it here for historical
purposes.

1.  Open a new xterm window. Click the left-button on the mouse in any
    open space on your window, then select ***New Shell*** to open an
    xterm window. It is prudent to run ***setup d0online*** once it is
    open -- it sets up what you will need, and it never hurts.

2.  Start the programs. In the xterm window type ***start_daq
    cal_examine***. Now type ***init*** \<carriage return\>, and
    ***start*** \<carriage return\>. That will start reading the data
    (which you can see flowing), and now you need to plot it. In another
    xterm window start up Histoscope by starting with ***setup
    d0online***, followed by ***histo &***. That pops up a window in
    which you can connect to the data by selecting the
    ***File/Process*** pull down menu and selecting ***Connect to
    process...***, that will load in something like ***Calorimeter
    Examine Histograms (d0cal)xxxx***. Highlight/select that file and
    left click the ***Connect*** button. Now you should see in the
    ***Sub Categories*** list that there is a single file selected, if
    so, left click on the ***Open*** button. A whole set of histogram
    names will appear in the right-hand pane.

3.  Choose one or a set of histograms that you would like to monitor. If
    you want to see a single histogram, then highlight it and left click
    on the ***View*** button. If you want to look at multiple
    histograms, select them (either by holding the shift key and
    clicking on a starting one and then and end one, or select them
    individually by selecting them with a Ctrl-click), and left click on
    the ***View Multiple*** button.

4.  Adjust the axes etc. Once you display the histograms, you may want
    to adjust the axes, select log scales etc. For example if you want
    to change the axis range, left click on the numbers on the axis and
    drag them. Many things can be done. You can check the users manual
    for more details (see
    <http://www-pat.fnal.gov/histo_doc/histo_ug.html>).

5.  Writing the data to a file. The histograms that you were looking at
    on the fly can also be saved to a file. That is done automatically
    as long as you stop the program correctly. In the window that shows
    the data being taken type ***stop*** \<carriage return\> and then
    ***quit*** \<carriage return\>. That will stop the data taking and
    also save the histograms to
    ***/online/examines/output/cal_examine_xxx.hbk*** where ***xxx*** is
    probably some random number like the process number (hopefully it
    will be clear which is the file you saved).

## Histoscope Error messages

***Error listing file: Error scanning /tmp directory***:

You may get this error when trying to connect to process. This can occur
because you haven't given the job long enough to collect sufficient data
(it needs about 15 events before you can connect), or it can occur
because of problems with temporary files (the detailed cause is unknown,
but there is a fix). In the second case, you will need to type ***setenv
TMPDIR /tmp/dummy_name*** (yes it means ***dummy_name*** exactly) in
both the xterm windows from where you are running examine (or the
cal_elec that is producing the histogram) and from where you are running
histo. Now try it again.

[]{#__RefHeading___Toc59549943 .anchor}Monitoring and Control Software

# Overview

There are a number of programs that allow you to monitor and interact
with the calorimeter hardware. For example you can "kill" cells (which
means making the window so large that the cell will never be read out),
or you can start, control and stop the pulser, you can turn on and off
power supplies (both low voltage and high voltage).

# Calorimeter Global Monitor

## Overview

## GUI Features

# Crate Monitoring

## Overview

## GUI Features

# Calorimeter Pulser

## Overview

The Calorimeter Pulser GUI is designed to allow you to control the
calorimeter pulser via the pulser interface board (PIB). It only needs
to be used during calibration and debugging. The latest version of this
GUI is Version 2 (the manual for version 1 can be found at
<http://www-d0.fnal.gov/~hohlfeld/pls_old.html>). The GUI was designed
by [marc_hohlfeld](#marc_hohlfeld) and current documentation is
available online at
<http://www-d0online.fnal.gov/www/groups/cal/pulser_control.html>. There
are 12 pulser modules for the Calorimeter and 1 for the ICD. In turn,
each pulser module has up to 96 channels that can be pulsed, which are
in turn fanned out to the individual preamps. The pulse height (actually
a current) can be adjusted by setting the DAC value. You also have
control of other more advanced features such as the delay, relative to
the trigger; of the command signal that allows groups of 16 pulse
channels to be sent to the preamps; the ability to step through a series
of delays and pulse heights automatically; and the option to select
specific pulser patterns to fire. In other words, the DAC value and the
command signal must be enabled for a pulse to make it to the preamp.

## Starting the Pulser GUI

To start the GUI, first type ***setup d0online*** (if you have not
already done this), then change directory by typing ***cd
/online/config/cal/pulser*** and ***./pulser.py***. The GUI ***Set
Calorimeter Pulsers*** will pop up (see Figure 3) . The ***PIB Clock***
in the top line of the GUI should update every second. If at some point
the update stops either the GUI has died or there is something wrong
with the PIB. You can change the update rate by clicking on the PIB
Clock field.

![](media/image2.jpeg){width="5.459027777777778in"
height="4.489583333333333in"}

Figure 3: Set Calorimeter Pulsers GUI

## Downloading a pulser

Here you will select the pulser or pulsers you wish to download, choose
a pulser amplitude setting (the DAC value), choose the delay in the
pulser signal relative to the trigger pulse, select the individual
channels you want to pulse, and download that configuration into the
PIB. To do so, follow these steps:

1.  Choose the pulsers you wish to you want to download. Click on the
    appropriate buttons in the ***Pulser Address*** section (upper
    left), when it turns red, you have selected that pulser. (0-11
    corresponds to the CAL, and 12 to the ICD). If you want to download
    all pulsers (12 CAL + 1 ICD) click on the ***All*** button.

2.  Enter a DAC value. You can type in a value from 0 to 131071 in the
    DAC Value box. (middle top).

3.  Enable the pulser channels. Select which of the 96 pulser channels
    (0-95) you want to enable (red box) by clicking on the appropriate
    button in the ***Channel Enable*** section. Note that you can make
    global selections by clicking on the ***Select All*** button in the
    ***Global Set Buttons*** section. You may have to toggle that button
    twice to make the selection.

4.  Enter the desired delays. You can enter a delay value of 0 to 255
    for each of six sets of channels (enter your value in the
    ***Delays*** section). Again you can set all six of them at once by
    entering a value in the ***Delay All*** box in the ***Global Set
    Buttons*** section.

5.  Set which commands are enabled. Recall that although you have
    selected which channels (0-95) will have a current setting
    corresponding to the pulse height. These channels will not actually
    pulse unless the command line, ***Cmd***, is enabled. There are 6
    command lines that control sets of 16 channels. Again, for your
    convenience, you can enable all 6 command boxes at once by setting
    the ***Cmd All*** button in the ***Global Set Buttons*** section.

6.  Download the values to the PIB. Click on the ***Download!*** button
    to send these values to the PIB.

That should complete the download of all the pulser settings to the PIB.
You should now be ready to take pulser data.

## Checking that the download was successful

Now that you have downloaded the pulser, you should check to make sure
that the download was successful before you actually use the pulser.

The colored boxes (0-B CAL, C ICD) in the ***Errors*** section (lower
right in the GUI show the error status for every pulser. The different
colors indicate the type of error:

-   purple: error was never checked

-   green: download without errors

-   red: an error occured during the download

-   yellow: somewhere between green and red, it usually means that there
    was no download to this pulser since the last reset of the PIB.

If an error occured you can see more details if you click on the
***Check Errors*** button. The ***Error Status*** window pops up. Enter
the pulser address (0-12) in the ***Pulser Address*** field and click
***Update***. The ***Error Message*** indicates the type of error (See
also \"Some other things you can do\" later in this documentation).

## How to make Global settings

It is also possible to make global settings of the basic selections for
the ***Pulser Address***, ***Channel Enable***, ***Delays*** and
***Cmd***. This is more convenient than making each selection
individually and can be a timesaver.

Make sure that the ***Select All*** button in the ***Global Set
Buttons*** section is on (red), and then:

1.  To enable/disable all 96 channels, click on the ***Enable All***
    button.

2.  To choose one value for all the six delays, enter the value in the
    ***Delay All*** field and press \"enter\".

3.  To enable/disable all six commands, click on the ***Cmd All***
    button.

## How to set a DAC/Delay Ramp

Just as you have done in the preceding sections, choose the channels,
delays and commands as you did earlier. Now you can set the range and
step size for automatically looping over a range of DAC settings and/or
a range of delay settings.

Click on ***DAC Ramp*** button and/or ***Delay Ramp*** button on the
right side of the GUI. Another window, ***Current Ramp Status***, will
pop up, as shown in Figure 4.

![](media/image3.jpeg){width="2.7083333333333335in" height="1.625in"}

Figure 4. Current Ramp Status GUI.

Some default values for number of steps, step size and events/step will
be automatically set in the appropriate fields on the right side of the
GUI. If you want, you can change these values by hand. Also the needed
request (65 for ***DAC Ramp***, 66 for ***Delay Ramp***) will be set in
the ***Request*** entry field. Do not edit this field by hand unless you
know exactly what you are doing!

To start the Ramp, click on the ***Download!*** button. The current step
and event should be updated in the ***Current Ramp Status*** window.

To stop the ramp, click on the ***Stop*** button in the ***Current Ramp
Status*** window.

## How to create a picklefile

Choose channels, delays, commands and ramp parameters as before.

Click on the ***PickleFile*** button.

The default filename is CAL_CCCP_PLSxx.pic, where xx indicates the
number of the pulser (00-15).

The default directory is /online/comics/cal (only write permission for
d0run!). If you are not logged in as d0run, choose a directory, where
you have write permission.

The layout of the picklefile is as follows (list of lists):\
\[\[word0, word1, word2, word3, word4, word5\],\
\[PrimaryAddress, WordOnOff, Steps, StepSize, EvSteps, Request\]\]

## How to choose a predefined pattern

Click on the \"Enable Pattern\" button. All channels and all commands
will be set to the off value and you cannot click on them anymore.

Enter the pattern number (0 to 31 are valid at the moment) in the
adequate field and press enter. Three channels and the three belonging
commands will be set.

## Some other things you can do

1.  Check errors: To see if the download was successful, click on the
    \"Check Errors\" button in the bottom line of the GUI. Another
    window (\"Error Status\") appears.

> ![](media/image4.jpeg){width="2.71875in"
> height="2.6041666666666665in"}
>
> It shows the pulser address, the pulser status and the error code. Do
> not worry about the error code, the error message should indicate what
> went wrong, if something went wrong. If an error occurs, the \"Error
> Message\" field turns red. After each download the window updates
> automatically, but if you want to check without a new download, enter
> the pulser address of the pulser you want to check in the \"Pulser
> Address\" field and click on the \"Update\" button.

2.  Read the current status of a pulser: Enter the address of the
    selected pulser in the \"Primary Address\" field and click on
    \"Update\".

3.  Shutdown: To turn off a selected pulser, insert the number of the
    pulser in the \"Primary Address\" entryfield. Click on \"Shutdown\",
    the DAC and Delay values are set to 0 and Commands and Enables are
    disabled.

4.  Save the current status of a pulser in a file: Choose the \"Primary
    Address\" of the pulser, read the current status of the pulser by
    clicking on \"Update\". Clicking on the \"Save\" button will pop up
    a new window. Enter the directory and the filename or browse through
    the file tree, then click on \"OK\".

5.  ![](media/image5.jpeg){width="4.916666666666667in" height="1.75in"}

6.  Read the status of a pulser configuration from a file: Click on
    \"Read\", choose your favorite configuration file and the
    appropriate directory and click \"OK\". Hopefully the GUI shows your
    selected configuration.

7.  Set the GUI to its default values: Just click on the \"Undo\"
    button.

![](media/image6.jpeg){width="6.370833333333334in"
height="7.415972222222222in"}

## 

## 

## GUI Features

# HV Control

## Overview

## GUI Features

# Hot Cell Killer

## Overview

The so called "hot cell killer" is designed to allow you to "turn off"
particular calorimeter cells so that they will not be readout. The
hardware accepts a pedestal value and a window. If the ADC value for a
particular cell falls within plus or minus the number of window channels
around the mean value, it will be suppressed and never read out. The
basic idea of the hot cell killer is that it downloads a very large
window value, hence assuring that the cell is never readout. It is based
on the latest pedestal sets which produce a new download (pickle) file
and a database entry. A backup of the previous version of the download
files is kept in case of fatal error.

The GUI also contains an expert mode which allows one to modify the
downloaded values. Of course nothing happens until the new values are
downloaded to the hardware. In order to carry out the hardware download,
you follow the usual procedure of using the cal_prepare_for_run trigger
file in Taker. The indeces of the current calibration sets used are
indicated at the top of each GUI page. At the bottom of each GUI page
are a number of display buttons. The DISPLAY HOT button shows all the
cells killed by the Hot Cell Killer in the current download file. The
DISPLAY BAD button lists all the cells killed by the regular calibration
validation. The DISPLAY MOD button lists all the cells that have been
modified in Expert mode.

The tabs at the top of the GUI are for: (a) the shifter to kill cells
(Shifter -- Kill); (b) the shifter to unkill cells (Shifter -- Unkill);
(c) the expert to modify the download values for a cell (Expert --
Modify).

## Channel Hierarchy

The hardware channel hierarchy is represented in the following diagram.
This structure is represented in the hot cell killer GUI.

![](media/image7.jpeg){width="4.420138888888889in"
height="2.845833333333333in"}

### Data Flow

A schematic view of how the hot cell killer script fits into the larger
picture of the pedestal calibration files and how the killed files enter
the download path.

![](media/image8.png){width="5.335416666666666in"
height="4.242361111111111in"}

## Step-by-Step Hot Cell Killing

Follow these steps to kill hot cells. This assumes you have found a set
of hot cell coordinates you want to kill.

1.  Start the Hot-Cell-Killer-GUI. Run the following script by typing
    the following: ***/online/comics/cal/New_HotCellKiller.py***

2.  Determine the crate/bls/adc/tower/depths coordinates. If you have
    used cal examine to identify hot cells, then the cells you want to
    kill are known in ***ieta/iphi/layer*** coordinates. You can convert
    that into hardware coordinates of ***crate/adc/bls/tower/depth***
    "manually" by running ***start_cal cpmap*** in a new xterm window to
    make the conversion. However, it is more convenient to use the
    second set of numbers generated by the hotcell finder report (the
    ***5 numbers in the 2^nd^ set of parentheses***) which is exactly in
    the ***crate/adc/bls/tower/depth*** coordinates you will need.

3.  Select the cell(s) to kill. First click on the ***Shifter Kill***
    tab in the ***hotcell killer GUI*** and ***select the boxes*** that
    contains the cell(s) to be killed. Note that you can click on more
    than one box in a row (you can clear your selection by clicking on
    the ***Deselect*** button). Once you are satisfied that you have the
    correct cells, then click the ***Save*** button and the selected
    channels will appear in the text window at the bottom of the page.
    You can keep doing this until you have selected all the cells you
    want. You can clear the list of cells by clicking the ***Reset***
    button. Once you have selected the cells, click on the ***Kill***
    button to produce a new download file and database entry. Finally
    follow the usual cal_prepare_for_run procedure to actually load
    these values in the hardware. In case of a database error, contact
    Ursula Bassler by email (<bassler@in2p3.fr>) or telephone (+33 (6)
    12 73 46 78).

4.  Or select the cells to unkill. It may be that there is good reason
    to "unkill" some cells. For example if you killed cells because of a
    bad BLS power supply which is now fixed. The procedure to unkill is
    similar to the kill procedure above except that you start by
    clicking on the ***Shifter Unkill*** tab in the ***hotcell killer
    GUI*** and ***select the boxes*** that contains the cell(s) to be
    unkilled. Note that you can click on more than one box in a row (you
    can clear your selection by clicking on the ***Deselect*** button).
    Once you are satisfied that you have the correct cells, then click
    the ***Save*** button and the selected channels will appear in the
    text window at the bottom of the page. You can keep doing this until
    you have selected all the cells you want. You can clear the list of
    cells by clicking the ***Reset*** button. Once you have selected the
    cells, click on the ***Unkill*** button to produce a new download
    file and database entry. Finally follow the usual
    cal_prepare_for_run procedure to actually load these values in the
    hardware. Note that the button called ***Unkill All Hot*** will
    restore all cells either killed or modified by the Hot Cell Killer
    since the last calibration was taken. In case of a database error,
    contact Ursula Bassler by email (<bassler@in2p3.fr>) or telephone
    (+33 (6) 12 73 46 78).

5.  Or modify the download values UNDER EXPERT GUIDANCE only. An expert,
    or you under their guidance, can modify the download values for a
    cell "by hand". First click on the ***Expert Modify*** tab in the
    ***hotcell killer GUI*** and ***select the boxes*** that contains
    the cell(s) to be modified, or select the unique ***channel-id
    number***. Click on the ***Retrieve*** button to get the pedestal
    values, zero-suppression limits and possible error-ids for this
    channel, and modify them as you wish. Once you are satisfied that
    the values are correct, click on the ***Save*** button; you will see
    this channel appear in the text window near the bottom of the page
    (you can click on the ***Clear*** button to remove all the channels
    you have saved if you need to). Repeat this procedure until you have
    modified all the channels you wish to change. Finally, click on the
    ***Commit*** button to produce a new download file and database
    entry with all the modifications you made. You will need to enter
    ***your name*** and a ***short comment*** before you can complete
    the commit procedure. In case of a database error, contact Ursula
    Bassler by email (<bassler@in2p3.fr>) or telephone (+33 (6) 12 73 46
    78).

6.  Download the new pedestals. The download can take 5-10min and nearly
    "locks-up" Coor, so for that reason, you ***must first request
    permission of the Shift Captain and inform the DAQ shifter that you
    would like to download the calorimeter pedestals*** -- they should
    care, and generally should give you priority for the download.
    Remember to ask the DAQ shifter to remove the calorimeter crates
    from any other runs that might be going. In the ***Taker*** click on
    ***Modify***, and then click on ***Change Trigger***. You will now
    see the configuration window. To make sure you are in the correct
    directory, click ***UP (Left)*** until you are all the way "up" and
    the directories no longer change. Double click on the directory
    ***commissioning***, then double click on ***cal***, then double
    click on ***cal_prepare_for_run-xxx*** (where xxx is the version
    number and if you find more than one version call the on-call
    expert). It will tell you ***download in progress***. If you get an
    error like **"Attributes conflict with already-owned item:
    crate:ecnnw. Request to coor failed."**, it probably means that some
    other run is using the calorimeter crates. Otherwise, the download
    takes a few minutes, and you will see lines like ***Done:
    crate:ecnnw*** written in the taker window. When you get to ***Done:
    crate:ecnne***, then you are done. Check with the DAQ shifter and
    ask him/her to remove the Cal crates from whatever running (global
    perhaps?) that is using them. Finally, if you know you don't have
    time to find hot cells again, then you should free the trigger by
    going to the Taker window, clicking on ***Modify*** and then
    clicking on ***Free Trigger***. 

# Channel Archiver

## Overview

The channel archiver is the program that captures and archives all the
information about the voltages, currents, etc for the calorimeter
subsystem. The present archiving for Calorimeter logs all channels every
5 minutes. At that rate the raw data files average about 600MegaBytes
for 7 days of running. It is therefore convenient to stop and start a
new archive every week. This section covers the procedures to follow to
maintain the archives. An archive in this context is actually a new
directory.

## Setup

As is usual, there are standard steps involved in dealing with the
channel archiver. You should start by opening an xterm window and typing
the following:

*\>setup d0online\
\>setup chan_archiver\
*

That will put you in the right directory and ready to issue archiver
commands. NOTE that these archives should be started and stopped by a
***d0cal*** user. Also the d0online and chan_archiver setup should be
completed

## Starting a new Archive

When you start a new archive, you essentially create a new directory.
The commands to do that are:

*\>cd /projects/archive/cal/current\
\>start_chan_archiver --c config_file --p 4814 --w .* (note the "." at
the end!)

## Stopping the Archive

In order to divide up the data into manageable pieces, we usually stop
and create a new archive on a weekly basis. It is a convenient time and
size (the data almost fills a CD).

*\>cd /projects/archive/cal/current\
\>stop_chan_archiver --p 4814*

## Remove previous data

We have been keeping the previous week of data in the /previous folder.
Before the new data can be moved to this area the folder must be cleared
out.

*\>cd /projects/archive/cal/previous\
\>rm --r \**

## Move Current data to previous

With the previous folder empty, the current data can be moved there with
this script\
\
*\>chan_arch_move cal*

## Determining if the Archiver is Running

Occasionally the archiver stops. It is useful to have a method to tell
if it is running. Here is how you can tell.

*\>ps auxww \| grep arch\
*

that will generate output that looks something like

d0rad 31924 0.3 0.2 11868 2532 ? S Sep25 4:52
/d0usr/products/chan_archiver/v1_9_1a/Linux/bin/ArchiveEngine config3

d0rad 31925 0.0 0.2 11868 2532 ? S Sep25 0:01
/d0usr/products/chan_archiver/v1_9_1a/Linux/bin/ArchiveEngine config3

d0rad 31926 1.4 0.2 11868 2532 ? S Sep25 19:58
/d0usr/products/chan_archiver/v1_9_1a/Linux/bin/ArchiveEngine config3

d0cal 14530 0.4 0.5 15332 5016 ? S 11:14 1:03
/d0usr/products/chan_archiver/v1_9_1a/Linux/bin/ArchiveEngine config4

d0cal 14531 0.0 0.5 15332 5016 ? S 11:14 0:00
/d0usr/products/chan_archiver/v1_9_1a/Linux/bin/ArchiveEngine config4

d0cal 14532 0.4 0.5 15332 5016 ? S 11:14 1:07
/d0usr/products/chan_archiver/v1_9_1a/Linux/bin/ArchiveEngine config4

In this example you see three d0cal processes running, which is normal
for the archiver.

## Restarting a Stopped Archive

If the archiver should stop (there various known and unknown reasons for
that), then you can restart this in the following way. First you need to
check if the lock file is present. If so, it must be removed.

*\>cd /projects/archive/cal/current\
\>rm archive_active.lck*

Now you need to restart the channel archiver using the directory name as
one of the command line parameters. For example, if the directory name
is dir.20011019-091652, then the commands are:

*\>cd /projects/archive/cal/current\
\>start_chan_archiver --c config_file --p 4814 --w . --d
dir.20011019-091652*

## Making a CDROM copy of the data

A copy of each weeks worth of data must be placed on a CDROM. A PC with
a CD burner and running WRQ (X-windows software with Kerberos) can be
used to FTP the data and place it on the CD. The only current machine
with FTP capability appears to be d0ol02.

## Viewing the Archive

The PC can view an archive using WinBrowser. Refer to
<http://d0server1.fnal.gov/www/online_computing/online_computing.html>
PPT, PDF, or PS: 4-Apr-2001-D0 Archiver Tutorial, Vladimir Sirotenko.
Sirotenko is the expert on Archiving.

[]{#__RefHeading___Toc59549981 .anchor}Downloading

[]{#__RefHeading___Toc59549982 .anchor}Calorimeter System Overview

# Preamplifier

# BLS

# Pulser

# ADC

# Timing and Control

# LaR Monitoring

[]{#__RefHeading___Toc59549989 .anchor}Preamp

# Overview

# Design

# Performance

# Details

# Safety

[]{#__RefHeading___Toc59549995 .anchor}BLS

# Overview

# Design

# Performance

# Details

# Safety

[]{#__RefHeading___Toc59550001 .anchor}Pulser

# Overview

# Design

# Performance

# Details

# Safety

[]{#__RefHeading___Toc59550007 .anchor}ADC

# Overview

# Design

# Performance

# Details

# Safety

[]{#__RefHeading___Toc59550013 .anchor}Trigger

# Overview

# Design

# Performance

# Details

# Safety

[]{#__RefHeading___Toc59550019 .anchor}Timing and Control

# Overview

# Design

# Performance

# Details

# Safety

[]{#__RefHeading___Toc59550025 .anchor}Rack Monitoring

# Overview

# Design

# Performance

# Details

# Safety

[]{#__RefHeading___Toc59550031 .anchor}High Voltage

# Overview

There are two high voltage systems -- one for the Calorimeter and one
for the ICD. This section covers both of those systems. You should
normally not have to play with the High Voltage. Once it is up, it
should remain up forever (although we have noticed that on occasion the
GUI stops updating -- so it looks like it is alive but it really isn't!
In that case you will notice that the small "propeller" on the title bar
stops turning). However there maybe be occasions when you need to start
them up after power outages or when it hangs. Most of this documentation
is taken from Alan Stone's web pages
<http://www-d0online.fnal.gov/www/groups/icd/controls/hv_control.html>.
Check there for the latest updates.

# Starting the GUI Software

There are actually two GUI's that are relevant. The first one provides a
global view of the HV state (indicating with colored boxes the state of
each channel, but not providing any details). It is used for monitoring.
The second one is arrived at by clicking in the first GUI, that will
pop-up a GUI that gives all the gory details about the HV, and it is the
one that is used to actually control the HV.

## Starting the Cal/ICD HV Monitoring GUI

If you need to start up the Cal/ICD HV GUIs to monitor the High
Voltages, follow these steps:

1.  Open an X-window. See the [open_xterm](#open_xterm) instructions
    [Error: Reference source not found](#open_xterm).

2.  Start the HV monitoring GUI. Type ***start_cal hv***. That will
    pop-up a monitoring window which is a representation of the HV racks
    in the first floor of the moveable counting house (M116 and M118).
    The colors indicate the state of the individual HV modules. Blue
    indicates that it is up and running at the set voltage. For more
    details see the following sections.

## Starting the Cal/ICD HV control GUI 

If you need to start up the Cal/ICD HV GUIs to reset or start up the
High Voltages, follow these steps:

1.  Start the HV monitoring GUI. See the steps above for starting the
    Cal/HV monitoring GUI. You will use this GUI to start the control
    GUI.

2.  Select the Cal or ICD control GUI. To open the control GUI,
    left-click on the title bar for each HV crate. For example to start
    up the ICD control GUI, ***left-click*** on the bar that says ***ICD
    North-East***. Or if you wanted to start up the Cal HV control GUI,
    then ***left-click*** on the title bar that for example says ***CAL
    North***. The GUI that pops-up will have four tabs on top that allow
    you to look at the details of each crate. It is from this GUI that
    you can reset or turn on and off the high voltages.

# Resetting the HV (ICD and Calorimeter)

If the HV trips, then you will need to reset it. Assuming that the HV
GUI is up and running (as it normally is), then follow these steps to
reset the HV:

1.  Select the tripped channel. The HV monitoring GUI (called the ***HV
    Global Monitor Display*** on the GUI label) will display a red bar.
    Left mouse click on the title bar of the "crate" that contains that
    red bar, and the HV control GUI will pop-up with the tripped channel
    again shown in red (this GUI is labeled ***HVC Channel Monitor***).

2.  Turn off ICD preamp power. If you are going to reset or turn on ICD
    HV, then you should first ***turn off the ICD preamps*** --
    otherwise the voltage spike may kill the ICD preamps. You turn off
    the ICD preamp power using the Calorimeter Power Supply Monitor
    Display GUI (= Supply GUI) . Go to the ***Preamp*** tab and near the
    bottom you will see ***ICD_LVCP_PW09***, and then click on the brown
    ***Off*** button on the right corresponding to that supply. This
    applies only to the ICD, there is no need to turn off Cal preamps if
    you are working with the Cal HV.

3.  Reset the HV. In this new HV control GUI, left-click on the tripped
    (red) cell. Left click on the cell, and a pop-up box will appear,
    select ***Reset***. The box will now turn orange and say ***On***.
    Note this does NOT mean that the HV is actually on, you must, in
    fact, left mouse click again and select ***Ramp***. The bar will now
    turn yellow, showing ***Ramp***, as the voltage ramps up. You can
    watch the voltage ramp up. Once it reaches its operating value
    (usually 2,000V) the box will turn green and show ***Holding***. At
    that point you want to lock the channel by left mouse clicking and
    selecting ***Lock***. Finally you should quit the GUI by left mouse
    clicking on the ***Quit*** button in the upper right hand side. If
    you don't succeed using this procedure, you might try increasing the
    voltage in stages. If that fails, call an expert. Make an entry in
    the logbook and if it is a calorimeter trip, send an email to
    [mailto:parua@fnal.gov?subject=Cal%20HV%20trip](mailto:parua@fnal.gov?subject=Cal HV trip),
    and if it is an ICD trip send it to
    [d0icd@fnal.gov](../d0icd@fnal.gov).

# Calorimeter High Voltage

The Calorimeter HV modules are located in the moveable counting house on
the 1^st^ floor. Here we describe how the HV modules are arranged. There
are 32 HV pods for each quadrant of the ICD (NE, NW, SE, SW). Each pod
sends voltage to a 1:3 fanout. There are 96 channels of HV for each
quadrant.

There are eight HV pods in each HV module, and there are six modules per
crate. The ICD uses 4x32 pods or 4x4 modules. The 16 modules are housed
on three crates (a, b and c) which sit in the rack MCH116.

Crate B controls power to all of the Northeast quadrant (pods NE01-32,
channels 1-96), and the first half of the Northwest (pods NW01-16,
channels 1-48). Crate C controls power to the second half of the
Northwest (pods NW17-32, channels 49-96) and all of the Southeast
(SE01-32, channels 1-96). Crate D controls only the power to the
Southwest (SW01-32, channels 1-96).

# ICD High Voltage

The ICD HV modules are located in the moveable counting house on the
1^st^ floor. There are 32 HV pods for each quadrant of the ICD (NE, NW,
SE, SW). Each pod sends voltage to a 1:3 fanout. There are 96 channels
of HV for each quadrant.

There are eight HV pods in each HV module, and there are six modules per
crate. The ICD uses 4x32 pods or 4x4 modules. The 16 modules are housed
on three crates (a, b and c) which sit in the rack MCH116.

Crate B controls power to all of the Northeast quadrant (pods NE01-32,
channels 1-96), and the first half of the Northwest (pods NW01-16,
channels 1-48). Crate C controls power to the second half of the
Northwest (pods NW17-32, channels 49-96) and all of the Southeast
(SE01-32, channels 1-96). Crate D controls only the power to the
Southwest (SW01-32, channels 1-96).

If you see something different than the above image, there is a problem.

For more information, check out the web link made by Alan Stone
<http://www-d0online.fnal.gov/www/groups/icd/docs/hv_map.html>.

In general, the calorimeter shifter should not modify the ICD HV
settings which have been painstakingly adjusted. There is a lot of
effort needed to adjust and calibrate the ICD signal, which is still
being defined and must be considered an Expert Task.

However, sometimes it will be necessary to turn off or on the ICD HV
because:

-   Work is being done in or near racks MCH116 or MCH117. The drip
    detector can accidentally be triggered by moving cables, so a
    controlled ramping down of the HV is preferable to a sudden and
    potentially harmful trip. The PMTs are quite old and fragile.

-   A cooling fan or power supply needs to be replaced, so the crate or
    rack needs to be powered down.

-   Water cooling or humidity problems may cause accidental RMI trips,
    so the HV needs to be ramped down as a preventative measure.

# Details

The racks MCH116, MCH117 and MCH118 are checked at the beginning of each
shift by the DAQ Shifter as part of the checklist procedure. This
includes making sure the Rack Monitor Interfaces are in a good state,
and that the cooling fans are all on.

The CAL-ICD HV monitoring GUI should already be open in the computer
monitor above the CAL/ICD console (d0oxt00). There are two columns each
with four blocks of LED cells in this monitoring GUI corresponding to
the HV crates housed in MCH116 and MCH118.

If the CAL-ICD HV monitoring GUI is not open, then while logged onto an
online account (d0icd, d0cal, d0run, etc.) type the following:

\> setup d0online\
\> cd /online/config/cal/hv/\
\> ./cal.hvg &

The following HV Monitoring GUI will pop up:

![](media/image9.jpeg){width="3.9555555555555557in"
height="5.111111111111111in"}

First, you need to open the ICD HV Control GUI.

Move your mouse cursor to the Crate panel on the HV Monitoring GUI. You
should see a little pop-up window with the phrase: Left click to display
channels. Do so.

The following HV Control GUI will pop up:

![](media/image10.jpeg){width="5.295833333333333in"
height="3.560416666666667in"}

There is tab for each of the four ICD quadrants, each powered by 32 HV
pods. It depends on which of the three Crate panels in the HV Monitoring
GUI you clicked which determines the tab you first see in the ICD HV
Control GUI.

Much of the ICD is not instrumented. To differentiate those channels,
they have been put in the Offline state.

How to turn off the HV:

1.  Go to the bottom and click on the button Unlock.

2.  At the top, click on Set HV and select 0%.

3.  Go to the bottom and click on the button Ramp.

4.  Never Turn Off any pods before ramping the voltage down to or near
    zero.

5.  Once all the values in the columns for V_Read and I_Read are at or
    very near zero, go to the bottom and click on the button Off.

6.  Do the above for all four quadrants, chosen by the tabs North-East,
    North-West, South-East and South-West.

How to turn on the HV:

1.  First ***turn off all ICD preamps***.

2.  Go to the bottom and click on the button On.

3.  In the NE, NW and SE, only some part of the ICD is instrumented.
    Therefore, it is better to only turn on the individual pods one at a
    time. Click on the cell with the left mouse button and choose On.
    You can tell which cells should be On by looking at the values in
    columns I_Max and V_Max. If the value is zero, then that cell should
    always be Offline.

4.  At the top, click on Set HV and select 100%.

5.  Go to the bottom and click on the button Ramp.

6.  The values in the columns for V_Read and I_Read should increase and
    the status cell will be yellow with the words Ramping or Average.

7.  When the status cell changes to Holding for all cells that are On,
    go to the bottom and click on the button Lock.

8.  Do the above for all four quadrants.

If you see in the monitoring GUI that some cell or cells have started
flashing red, this means that some HV pods have tripped, usually on
over-current.

How to reset the HV if it has tripped:

1.  Go to the ICD HV Control GUI (see above if you need to open one).

2.  Click through the four tabs to find the HV pods where the status
    cell is flashing red.

3.  Click on the cell with the left mouse button and choose Reset. If
    there are multiple trips, go to the bottom of the GUI and click on
    Reset. This should change the state to On.

4.  Go to the bottom and click on the button Ramp.

5.  The values in the columns for V_Read and I_Read should increase and
    the status cell will be yellow with the words Ramping or Average.

6.  When the status cell changes to Holding for all cells that are On,
    go to the bottom and click on the button Lock.

7.  Make a note in the logbook and send email to Alan Stone
    (alstone@fnal.gov).

I have setup up scripts to launch StripTools of all the ICD currents and
voltages. They are set with a circular buffer of some finite number of
seconds, so one can go back in time to see why something tripped. Both
scripts generate nine separate windows stacked on top of each other, so
you will need to space them out in order to view them all
simultaneously. The default window shows the last five minutes with a
resolution of one second. We are still trying to understand the
behavior. The currents read from the ICD HV SW crate have an oscillating
feature which appears to be real.

You need to be logged into d0ol45 (as d0icd, d0cal, d0run) Type the
following:

\> setup d0online\
\> cd /home/d0icd/hv/\
\> ./ICD_CurrentStripTool &

Nine stacked windows like the following will open:

![](media/image11.jpeg){width="4.159027777777778in"
height="2.6465277777777776in"}

There are six buttons on the bottom which allow you to: pan left; pan
right; zoom in; zoom out; auto scroll; refresh.

When you manipulate the pan or zoom functions, the StripTool may no
longer appear to be updating. This is not the case. The information is
still stored in buffer but not displayed. You can continue the updating
by clicking on the Autoscroll button.

The upper and lower limits for each channel are set in the script such
that increasing current draws that lead to a trip can be seen.
Alternatively, a trip may be caused by something very sudden. As long as
a trip can be spotted and recorded shortly after it happened, we can go
back and look in the StripTool.

To open the ICD voltage StripTools, type the following:

\> setup d0online\
\> cd /home/d0icd/hv/\
\> ./ICD_VoltageStripTool &

Only as long as the StripTools have been open can we look back in time
to see the current or voltage behavior. So, if you noticed that the
StripTools have been closed, please relaunch them. I have dedicated one
screen each to voltage and current. If memory or space is limited, I
would prefer that voltage be sacrificed first. They should also be added
to the start-up procedure for the Calorimeter/ICD console (d0ol45).

[]{#__RefHeading___Toc59550040 .anchor}LAr Monitoring

# Overview

# Design

# Performance

# Details

# Safety

[]{#__RefHeading___Toc59550046 .anchor}Appendix A -- Rack Maps

There are a number of locations where you will find electronic racks
that contain electronics relevant to the calorimeter system.

# On the Calorimeter

The term "rack" is used rather loosely defined for the locations on the
calorimeter. It is the location of the preamplifiers and their
associated power supplies.

## Preamps

## Preamp Low Voltage Power Supplies

## HV distribution

# Under the Detector

The signals from the preamps are transported via twist-and-flat cables
from the preamp boxes on the calorimeter to the crates that contain the
BLS cards.

## BLS

## BLS Low Voltage Power Supplies

# In the Moveable Counting House (MCH)

The signals from the BLS crates are sent via twist and flat cables from
the BLS racks to the ADC's and the L1 Cal Trigger located in the
Moveable Counting House (called MCH). The MCH has three floors.

## ADC

## ADC Low Voltage Power Supplies

## Timing and Control

## CETEC Low Voltage Power Supply

## HV Power Supplies

## L1 Cal Trigger

# In the Control Room

[]{#__RefHeading___Toc59550062 .anchor}Appendix B -- Safety
Documentation

[]{#__RefHeading___Toc59550063 .anchor}Appendix C -- Scripts Explained
(for Experts)

# Overview

In this appendix we capture some of the detail behind the scripts. It is
not intended that any of this information needs to be used by shifters,
but it is collected here so that when the corporate memory of the
scripts are lost, then there will be some documentation of what they are
supposed to do. Essentially this Appendix is created by moving
documentation sections from the front of the shifter manual to here --
as new scripts are developed.

# The Scripts

This is a somewhat haphazard collection of documentation that are
presently obsolete, but may prove useful.

## Downloading Pedestals 

The pedestal download is now done by the script **./download_peds.sh**.
Here are the steps that it essentially takes.

1.  Change the directory. In your xterm window, prepare the way by doing
    a ***setup d0online***. Then change the directory with ***cd
    /online/config/cal/comics*** and type ***su d0run*** (which has a
    password that you will have to find in the written documentation in
    the control room -- it is written on the front divider labeled
    Contact Information).

2.  Check to see if Comics is running. You can check to see if Comics is
    running with ***ps axx* \| *grep comics***. If it is running, you
    will see a four lines that look like

> *21653 pts/18 S 0:00 python ./comics.py*
>
> If it isn't running, then start it up with the usual ***setup
> d0online*** followed by ***./comics.py &***.

a.  Start the download. As usual, try ***setup d0online*** to start with
    (never hurts). Then run ***./talkcomics.py --f
    download_peds.txt***.. You will see it scroll through each of the
    crates starting with 13. There may be failures to load individual
    crates (watch the messages as they scroll by) -- if so, simply try
    running it again till it passes. Note that if it turns out that
    comics is not running, then you will see the error message
    ***Couldn't connect*** when you try this step -- go back in that
    case. There may also be failures to download individual crates
    (watch the messages as they scroll by, and if you see "n ok", where
    "n" is a crate number, then that crate has been downloaded). Since
    the script repeats four times, they should be properly downloaded by
    the time the script finishes running. If not, run it again. Note
    that once a crate is correctly downloaded, it stays that way, so
    each time you run the script you will eventually get all of the
    crates to download properly.

## Taking pedestal Calibration Runs

In the past, it used to be that you could only take calibration runs for
pairs of crates, so you would have to cycle through 6 pairs to do the
whole detector. Here are some of the steps we followed then. These are
just copied from obsolete steps from the front of the shifter guide.

1.  Select the trigger for the Taker and start the run. In the Taker,
    click on ***modify***, and then click on ***Change Trigger***. You
    will now see the configuration window. To make sure you are in the
    correct directory, click ***UP (Left)*** until you are all the way
    "up" and the directories no longer change. Double click on the
    directory ***calibration***, then double click on ***cal***, then
    double click on ***calib-0X40-0X41-1.0***. It will tell you
    "download in progress". After the download is complete (about 30
    seconds) click on ***start*** (lower left long button in Taker) to
    begin the run. A pop-up ***Begin Run*** window will appear. Fill it
    in with your name in the ***shifter*** box, the gain (x1 or x8) and
    the crates being run in the ***Reason*** box, and check the
    ***Calibration*** button. That box disappears and the run has
    started.

2.  Repeat for all other crates. You now go back to the taker and repeat
    this same procedure for the other remaining crates, starting with
    step Error: Reference source not found [Error: Reference source not
    found](#select_trig_for_taker) (the files are called
    calib-0x42-0x43-1.0 and so forth; it should be obvious). That means
    you will modify the trigger file appropriately. When you have run
    through all crates (6 files) you have completed the pedestal
    calibration run for the x1 gain.

## Checking the memory on the T&C board after it is power cycled

This is a step that can now be carried out through the CRATE MONITORING
GUI (=T&C GUI), but it is included here for archival purposes.

1.  Check the memory address 0x40ff0088 is set to 0x88b. If it is not
    set, then you need to set it. The amber SCL synchronization light on
    the SCL receiver card should now be on. If you need to get to the
    VME memory, then you need follow these directions:

> Log into any online linux machine or d0ola or d0olb, and type

-   setup onl_apps

-   cd /online/config/cal

-   ./vme_xx & (where xx is crate 40 to 4A)

> A window will pop up; click on the left side of up to four panes in
> the window. Another pop-up window will appear, change the desired
> value and click on the ***Apply*** button.

## Make sure we are running in the correct mode for data taking 

These instructions allow you to set the electronics into the correct
mode for data taking. This is now all taken care of by the download
called "cal_prepare_for_data_1.1".

You can check this in the Calorimeter Power Supply Monitor Display GUI
(= Supply GUI) by going to the ***Mode*** tab (the last one on the
right). Check the first purple column labeled ***MODE***. It should show
***0x80f*** for all 12 crates. If it showing anything else, you will
need to change it by starting up the ***Crate Monitor GUI*** (more
commonly referred to as the Timing and Control or CRATE MONITORING GUI
(=T&C GUI)). If you need to open this GUI, run ***start_cal tandc*** in
your open xterm window. Click on the ***Global BLS Mode*** button and
click ***Normal*** on the drop down menu. Now close the menu by clicking
the ***Global BLS Mode*** button again. Finally click on the ***Global
ADC Mode*** button and then click on the ***0SignSup*** button. Close
the drop down menu by clicking the ***Global ADC Mode*** button again.
Click on the ***Global T&C set*** , and click on ***x88b***, which
corresponds to the so-called "fixed bunch" mode. You can check this by
looking at the ***Calorimeter Power Supply Monitor Display*** and
selecting the ***Mode*** tab -- the first column (of yellow/beige
color), ***MODE***, should show the corresponding value ***0x88b*** for
all 12 crates. Finally, you should check to see that the Cell and Tick
settings (which may have been changed for pulser running) are correct --
that means that the 4^th^ yellow/brown column in the Pulser GUI called
***DIAG*** should read ***0x0***. If it is not 0x0, then go to the CRATE
MONITORING GUI (=T&C GUI) and click on the ***Global T&C Set*** button,
click on the ***CELL*** box and type in ***0***, and then on the
***TICK*** box and type in ***0***. There are also two columns to the
left of the yellow reset buttons on the right. Those two columns should
be green and labeled ***Normal*** and ***0 Sign Sup***. If not, go back
a few lines and make sure you followed instructions.

## Set the Timing and Control system to fixed cell mode, 0x889

To do this, go to the ***Crate Monitoring*** GUI (called the CRATE
MONITORING GUI (=T&C GUI)). If it isn't open, go to your xterm window
and run ***start_cal tandc***. Click on the purple button, ***Global T&C
Set***, and select ***0x889*** and enter ***f0*** in the ***CELL*** box.
The ***MODE*** column on the Calorimeter Power Supply Monitor Display
GUI (= Supply GUI) (the first orange/beige column) should now show
***0x889 (fixed cell)***.

## Starting the HV GUI's

This is now done via a single script ***start_cal hv***. Here are the
pieces that go into that script.

1.  Set the directory and setup. First type ***setup d0online*** and
    then change the directory by typing ***cd /online/config/cal/hv***.

2.  Start the calorimeter HV GUI. Enter ***./cal.hvg &*** .

## Run the HISTO version of Calorimeter Examine 

We have switched over to the Root version of Examine, but in the past we
ran the histo version. Here is how we did that. Open a new xterm window
(on the left screen usually). Start with the usual ***setup d0online***,
then, run ***start_daq cal_examine***, on completion, follow by
***init*** \<carriage return\> and ***start*** \<carriage return\> (see
page [160](#running-the-histoscope-version-of-the-software) for more
details). That will start reading the data, and now you can plot it. In
another xterm window start up Histo by starting with ***setup
d0online***, followed by ***histo &***. That pops up a window in which
you can connect to the data by selecting the ***File/Process*** pull
down menu and selecting ***Connect to process***..., that will load in
something like ***Calorimeter Examine Histograms (d0cal)xxxx***.
Highlight/select that file and left click the ***Connect*** button. Now
you should see in the ***Sub Categories*** list that there is a single
file selected, if so, left click on the ***Open*** button. A whole set
of histogram names will appear in the right-hand pane. If you want to
see a single histogram, then highlight it and left click on the
***View*** button. If you want to look at multiple histograms, select
them (either by holding the shift key and clicking on a starting one and
then and end one, or select them individually by selecting them with a
Ctrl-click), and left click on the ***View Multiple*** button.
Eventually we will have a set of reference Examine plots for you to
compare with. For now just keep an eye out for anomalies, such as new
and unusual structures in the distribution of hits, etc.

1.  Run the Root Calorimeter Examine. Examines should be run on d0ol23
    node which is faster and relieves the load from d0ol45. If there is
    an Examine GUI open, then you are done. If not, the open a new xterm
    window (on the left screen usually), type the usual ***setup
    d0online*** and then ***d0ssh d0ol23***. It used to be -- maybe not
    necessary now? \-- that you should now check to see if any examines
    or xgooey processes are running by typing ***ps --ef \| grep
    xgooey*** and then ***ps --ef \| grep examine***. These instructions
    should list all the processes that contain ***xgooey*** in their
    name, if for some reason you want to see ***all processes*** then
    type ***ps --fu d0cal*** -- this should not be necessary, but
    shifters have reported problems with the first instructions. Kill
    those processes (i.e. any processes that contain ***xgooey*** in
    their name, like ***xgooey_xx*** etc) using ***kill -9 xxx*** where
    ***xxx*** is the process number. Then ***setup d0online*** and run
    ***start_cal examine***. This opens the ***Examine Control Panel***.
    Click on the ***Edit*** button and select ***Preferences***. Select
    the ***EXAMINE*** tab and set ***Server Address*** to
    ***d0ol23.fnal.gov*** and check the ***Canvas Menu Bar*** option
    under the ***DISPLAY*** tab. Back on the main control panel, click
    the red ***Start*** button (upper left), which will bring up a list
    of histograms. Choose one or a set of histograms that you would like
    to monitor by selecting them with a left mouse click. Now click on
    the light-green ***Display*** button in the ***Histograms*** menu
    (which is at the bottom of the ***Examine Control Panel***.). That
    will pop open a window with those histograms. Don't forget to
    unselect the histograms before opening a new display (or "canvas" in
    root language), otherwise you will see them again! You can play with
    adjusting sizes, scales etc. Keep the number of histograms displayed
    to as small a number as possible. The system tends to slow down or
    hang otherwise. You can report problems in an "observation file"
    which you can copy (with xemacs) into the logbook.

2.  For now, run the Histoscope Examine version. At some point this will
    be the Root version, but not just yet. You should monitor Examine
    throughout the run -- starting a fresh Examine at the start of the
    run, and stopping at the end of a run. To run the Examine, open a
    new xterm window and run :

> *\> setup d0online*
>
> *\> d0ssh d0ol23*
>
> *\> setup histo*
>
> *\> histo* &
>
> *\> setup d0online*
>
> *\> cd /online/examines/scripts/*
>
> *\> ./cal_examine.sh*
>
> *init*
>
> *start*
>
> When you ran the histo command then it popped up a window in which you
> can now connect to the data by selecting the ***File/Process*** pull
> down menu and selecting ***Connect to process***..., that will load in
> something like ***Calorimeter Examine Histograms (d0cal)xxxx***.
> Highlight/select that file and left click the ***Connect*** button.
> Now you should see in the ***Sub Categories*** list that there is a
> single file selected, if so, left click on the ***Open*** button. A
> whole set of histogram names will appear in the right-hand pane. If
> you want to see a single histogram, then highlight it and left click
> on the ***View*** button. If you want to look at multiple histograms,
> select them (either by holding the shift key and clicking on a
> starting one and then and end one, or select them individually by
> selecting them with a Ctrl-click), and left click on the ***View
> Multiple*** button. Eventually we will have a set of reference Examine
> plots for you to compare with. For now just keep an eye out for
> anomalies, such as new and unusual structures in the distribution of
> hits, etc. You must look at least at the set of plots for Nhits vs
> ieta-iphi, and Energy vs ieta-iphi. If you find a "tower" that is
> abnormally high (or low) compared to its neighbors, then you should
> cycle through the similar plots for the individual layers, in order to
> identify the problem cell. Record your findings in the electronic
> logbook (try pasting in a copy of the plot in the logbook).

Run the Root Calorimeter Examine. It is broken at the moment. Examines
should be run on d0ol23 node which is faster and relieves the load from
d0ol45. If there is an Examine GUI open, then you are done. If not, the
open a new xterm window (on the left screen usually), type the usual
***setup d0online*** and then ***d0ssh d0ol23***. Now check to see if
any examines or xgooey processes are running by typing ***ps --ef \|
grep Xgooey*** and then ***ps --ef \| grep examine***. Kill those
processes using ***kill -9 xxx*** where xxx is the process number. Then
***setup d0online*** and run ***start_cal examine***. This opens the
***Examine Control Panel***. Click on the ***Edit*** button and select
***Preferences***. Change ***EXAMINE*** to ***d0ol23.fnal.gov*** and
check the ***Canvas Menu Bar*** option under ***DISPLAY***. Back on the
main control panel, click the red ***Start*** button (upper left), which
will bring up a list of histograms. Choose one or a set of histograms
that you would like to monitor by selecting them with a left mouse click
(at the moment ***do not select Total Energy, Total Et, EM energy, Had
Energy, EM cell energy ieta/iphi, HAD cell energy ieta/iphi***). Now
click on the light-green ***Display*** button in the ***Histograms***
menu (which is at the bottom of the ***Examine Control Panel***.). That
will pop open a window with those histograms. Don't forget to unselect
the histograms before opening a new display (or "canvas" in root
language), otherwise you will see them again! You can play with
adjusting sizes, scales etc. Keep the number of histograms displayed to
as small a number as possible. The system tends to slow down or hang
otherwise. You can report problems in an "observation file" which you
can copy (with xemacs) into the logbook.

3.  Monitor Examine plots during the run. Once you have selected the
    plots of interest, you can update the histogram to include more
    statistics by clicking the ***Update*** button. Note that at each
    update all the settings for the histogram are lost and it is
    returned to the default settings. You should try and start a fresh
    set of histograms for each "significant" run (you be the judge of
    what is significant). You can check with the DAQ shifter to see when
    runs start and stop. Once you have checked the plots for a run, and
    noted the various "hot" regions (if any) in the Runs Database, then
    you can check the ***Online Checked*** option in the runs database.
    If you want to place a screenshot of a histogram in the logbook, you
    can follow the directions on page [59](#_Ref4211357).

Set the BLS/ADC mode to 0xb01. Go to the CRATE MONITORING GUI (=T&C
GUI). If it isn't already open go to an xterm window and run
***start_cal tandc*** (if you have trouble, check the troubleshooting
guide). Click on the purple ***Global BLS Mode*** button, and then click
on ***Force x8***. Remember to close the pull down menu by clicking once
more on the same ***Global BLS Mode*** button. Now click on the purple
***Global ADC Mode*** button and click on ***Unsup*** (for
unsuppressed). That should have changed the ***MODE*** column (first
purple column) on the Calorimeter Power Supply Monitor Display GUI (=
Supply GUI) to ***0xb01***. You can also check to see that the change
was carried out by checking the Calorimeter Power Supply Monitor Display
GUI (= Supply GUI) 's ***Mode Shift*** tab, where you will see two blue
columns (labeled ***BLS MODE*** and ***ADC MODE***) indicating ***Force
x8*** and ***Unsuppressed***. Under normal data taking conditions these
two columns are green.

What the pedestal calibration does. (for example while the pedestal
calibration is running, it checks for cells with pedestal values less
than 400 and greater than 800 or a zero-suppression threshold greater
than 100 for the x8 gain path), and those cells are suppressed by
putting a zero-suppression threshold of 32760 in the pedestal download
file (/online/comics/cal/calib/data/crate0x%%-download.txt where %% is a
crate number from 40 to 4b). All the cells that are suppressed are
listed in /online/comics/cal/calib/valid/crate_all_bad.txt.

## If the FPGA code needs to be downloaded into the T&C crate

If the T&C board power has been cycled, you will need to reload the code
into the FPGA's. To do this involves finding the ADC racks on the 3^rd^
floor of the movable counting house, plugging in and moving a cable.
These instructions should allow you to do that, but it is best left to
an expert unless it is an emergency.

1.  Locate the download computer in MCH309. The PC used to download the
    FPGAs is located on the third floor of the movable counting house
    (MCH) in rack 309 (the racks are labeled on top). Open the door in
    the rack to access this NT PC. If the PC was rebooted, you may need
    to log in as ***d0cal*** with the same password for d0cal as for the
    online (see the password written on the front page of the user guide
    binder).

2.  Start up the FPGA software. On the desk you will find the
    ***max+Plus2*** icon, double click on it. The ***Altera maxplusII***
    window should pop up. Click on ***Max+plusII***, then
    ***Programmer***. Another window pops up. Now leave it while you
    attach the cables.

3.  Attach the Programming cable to the fanouts. Take the 8-pin ***FPGA
    Programming cable (the blue one,*** or is it ***grey with a blue and
    red stripe?)*** and plug it into the ***smaller*** of the two
    ***fanout*** boards (the one on the right). There are two fanout
    boards which are located in the small section between the top
    electronics crate and the one below it, in rack ***M308***. The
    connector is marked by a ***white stripe that faces left*** (and
    matches the white mark on the fanout board connector), and it plugs
    into the ***leftmost male connector*** on this smaller one of the
    two fanout boards -- the small fanout board is on the left, and the
    larger one on the right. Now verify that the cable labeled ***12***
    is plugged into the ***4^th^ open connector*** (counting from the
    right) on the small fanout board. It should be the only "open"
    connector (i.e. one without a little two pin jumper in it). Be very
    careful not to pull on the cables going to the ADC boards, as they
    detach from their connectors on the boards quite easily.

4.  Download the files to the FPGA. Go back to the computer now, and
    pull down the ***JTAG*** menu in the ***Max+plusII*** window, select
    ***Restore JCF***, and browse through ***Directories*** starting at
    ***maxplus2*** then ***work*** then ***tandc_ctrl***. In that
    directory select ***tandc_ctrl.jcf*** by double-clicking on it. At
    that point, the programmer window displays ***Multi Device JTAG
    Chain*** (4 programming files). Click on ***Configure***. You will
    see activity indicating that the files are being downloaded. When it
    is done, you will see the message ***Configuration complete: 4
    devices configured***. If you do not see the complete message, or if
    there is an error like ***jtag information mismatch***, it is most
    likely due to the cable not being plugged in correctly, or perhaps
    upside down. Correct the problem and try again.

5.  Move the Programming cable and repeat for the large fanout board for
    cables 0 to 5 -- CAREFUL, these next three steps have a tendency to
    fail, if so, skip them and go to step 8. Remove the blue programming
    cable from the small fanout, and move it to the large fanout, where
    you will again plug it into the rightmost connector (and again the
    white mark is on the left and matches the one on the fanout
    connector). Verify that the cables labeled ***0*** through ***5***
    are plugged in in sequence to all the open connectors on the large
    fanout (the open connectors are the ones without the 2-pin jumpers
    in them).

6.  Download the files to the FPGA. Go back to the computer now, and
    pull down the ***JTAG*** menu in the ***Max+plusII*** window, select
    ***Restore JCF***, and browse through ***Directories*** starting at
    ***maxplus2*** then ***work*** then ***tandc.*** In that directory
    select ***tandc6.jcf*** by double-clicking on it. At that point, the
    programmer window displays ***Multi Device JTAG Chain*** (18
    programming files). Click on ***Configure***. You will see activity
    indicating that the files are being downloaded. When it is done, you
    will see the message ***Configuration complete: 18 devices
    configured***. If you do not see the complete message, or if there
    is an error like ***jtag information mismatch***, it is most likely
    due to the cable not being plugged in correctly, or perhaps upside
    down. Correct the problem and try again. Note however that after
    power cycling of the crate it is possible that you may not be able
    to program the FPGAs using the large fanout card. In that case skip
    the next step and go to the one after.

7.  Change cables 0-5 with cables 6-11. Now remove cables ***0-5*** from
    the large fanout board, and plug in cables ***6-11*** that should be
    dangling nearby. Click on ***Configure***. You will see activity
    indicating that the files are being downloaded. When it is done, you
    will see the message ***Configuration complete: 18 devices
    configured***. If you do not see the complete message, or if there
    is an error like ***jtag information mismatch***, it is most likely
    due to the cable not being plugged in correctly, or perhaps upside
    down. Correct the problem and try again.

8.  If you cannot program using the large fanout, which is likely! If
    can succeeded in programming with the large fanout board, then skip
    this step. Otherwise, as was mentioned above, it is possible that
    after a power cycle you will be unable to program the FPGAs using
    the large fanout card. In that case you will have to fall back to
    the mode of programming using the small fanout and programming one
    card at a time. Sorry. So, in this case start by plugging in cable 0
    into the one open slot in the small fanout card (of course don't
    forget to plug in the blue programming cable into that rightmost
    slot!). Go back to the computer now, and pull down the ***JTAG***
    menu in the ***Max+plusII*** window, select ***Restore JTAG***, and
    browse through ***Directories*** starting at ***maxplus2*** then
    ***work*** then ***tandc.*** In that directory select
    ***tandc.jcf*** by double-clicking on it. At that point, the
    programmer window displays ***Multi Device JTAG Chain*** (3
    programming files). Click on ***Configure***. You will see activity
    indicating that the files are being downloaded. When it is done, you
    will see the message ***Configuration complete: 3 devices
    configured***. If you do not see the complete message, or if there
    is an error like ***jtag information mismatch***, it is most likely
    due to the cable not being plugged in correctly, or perhaps upside
    down. Correct the problem and try again. Now repeat one at a time
    for the remaining cables ***1-11***.

9.  Exit the program. Click ***File***, then select ***Exit***.

10. Verify that you have entered the correct FPGA code version. The
    Calorimeter Power Supply Monitor Display GUI (= Supply GUI) has an
    entry for that version number (presently 27), which you should check
    to see that it is correct. Also, the T&C controller mode should be
    set according to step Error: Reference source not found on page
    [Error: Reference source not found](#_Ref5158354).

[]{#__RefHeading___Toc59550074 .anchor}Appendix D - Calorimeter
Checklist

The calorimeter checklist is now just part of the combined
calorimeter-muon checklist which is available in the electronic logbook.

[]{#__RefHeading___Toc59550075 .anchor}Index

A

ADC, 7, 12, 37, 39, 40, 42, 43, 65, 68, 69, 88, 92, 110, 113

Altera, 113

Archiving, 27

B

BLS, 7, 10, 15, 32, 34, 37, 39, 40, 41, 42, 43, 48, 65, 68, 69, 88, 90,
110, 113

C

, 10, 34, 67, 69, 70, 71, 110, 111, 112

, 10, 11, 22, 72, 109, 113

Captain, 23, 39, 43, 60

Connect, 9, 10, 34, 51, 71, 110, 112

, 10, 37, 39, 40, 43, 65, 67, 68, 69, 98, 99, 108, 109, 110

D

DAQ shifter, 23, 34, 39, 43, 99

Delay ramp, 38, 39

, 8, 11, 16, 24, 25, 26, 30, 34, 36, 37, 39, 40, 42, 43, 44, 51, 65, 67,
68, 69, 71, 73, 76, 77, 78, 83, 108, 109, 110

, 5, 18, 22, 107

Download, 8, 12, 34, 39, 40, 43, 44, 65, 73, 74, 76, 77, 83, 108, 109,
113

E

Edit, 25, 27, 67, 76

Errors, 4, 34, 39, 43, 50, 51, 71, 74, 75, 76, 77, 108, 114, 115

, 10, 34, 67, 69, 70, 71, 110, 111, 112

Expert, 4, 11, 14, 26, 59, 68, 97, 113

F

FPGA, 113, 114

G

Gain, 10, 12, 37, 39, 40, 42, 43, 109, 113

H

, 65, 79, 96, 97, 98, 99, 100, 101, 102, 103, 110

, 34, 51, 69, 70, 71, 110

, 34, 51, 69, 70, 71

, 9, 10, 17, 32, 79

L

Log on, 23, 30, 60, 109

Logbook, 6, 10, 14, 16, 27, 37, 39, 40, 42, 43, 97, 102

M

Max+, 113

N

New shell, 23

P

passwords, 6, 108, 113

Passwords, 6, 22, 23, 108, 113

Pedestal, 8, 10, 15, 37, 39, 40, 42, 43, 59, 65, 108, 109, 113

PLS mode, 8, 38, 39, 40, 43

Preamp, 7, 34, 56, 65, 68, 69, 89

, 34, 56, 65

Printing, 24, 27

Programming, 113

Pulser, 7, 8, 9, 36, 37, 38, 39, 40, 42, 43, 72, 73, 74, 75, 76, 77, 78,
110

Pulser Interface Board (PIB), 38, 39, 40, 42, 72, 73, 74

R

, 65

Reboot, 23, 60

, 9, 10, 14, 69, 111, 112

, 34, 50

Root, 69, 70, 111, 112

Run, 4, 6, 8, 10, 11, 12, 16, 23, 24, 30, 31, 32, 34, 36, 37, 38, 39,
40, 42, 43, 56, 65, 66, 67, 68, 69, 70, 108, 109, 110, 111, 112, 113

S

SAM, 30

, 34, 50, 56, 65

Stop, 16, 34, 37, 39, 40, 41, 42, 43, 71, 76

, 10, 34, 38, 39, 40, 43, 56, 110, 113

T

T&C GUI, 15, 34, 37, 39, 40, 41, 42, 43, 109, 110, 113

Table of contents, i

Taker, 11, 16, 26, 34, 36, 37, 39, 40, 42, 43, 65, 83, 109

Timing, 37, 39, 43

, 39, 42, 65, 72, 73, 109

Troubleshooting, 4, 34, 49, 113

X

, 9, 11, 14, 24, 31, 34, 37, 38, 39, 40, 42, 43, 50, 51, 56, 60, 65, 70,
71, 108, 110, 111, 112, 113

A Xterm, 11, 12, 29, 30, 36, 42, 45, 47, 50, 51, 53, 55, 56, 57, 58, 59,
60, 61, 63, 65, 67, 71, 72, 78, 79, 80, 85, 86, 123, 125, 126, 127, 128