# Presentation: 557404

## DØ SMTShift Instructions

- Version 1.41

## Contents

_**Page**_**  **_**Section**_
- General Instructions
- Logbook
- Operations Database
- All-GUIS-starter
- DVDD GUI
- Low Voltage Power Supplies
- IB Temperature GUI
- Utility Scripts and GUIs
- Listing Scripts and GUIs
- Rack Monitor Interface
- IOCs
- Download GUI
- SDAQ/Calibrations
- PDAQ/Crater/Taker
- High Voltage
- Alarm Display
- StripTool
- SMT Event Display
- SMT Examine
- Occupancy GUI
- Monitoring
- Archivers
- MCH Walkthrough
- Radiation Monitoring
- Paging

## General Instructions

_**Page**_**  **_**Section**_
- Shift Guidelines
- Overview of the detector
- HDI types
- Schematic diagrams
- LV and HV
- Download parameters
- High DVDD current and LV trips
- Increasing HV current limits
- Monitoring

## Shift Guidelines

- If you haven’t done so, read the instructions!
- Login into the logbook at the beginning of your shift.
- Log everything you do, and try to be thorough in the description of problems you may run into. Fill in the checklists relevant to what is going on. Start filling in the « Begin of Shift » checklist as soon as you start your shift.
- Most of the GUIs you need can be started from the « GUI starter » GUI (in a terminal window type: cd; setup d0online; ./startguis.py)
- If one of the VRB crates is 100% FEB try the following, in order, until the busy is cleared:
- Ask the DAQ expert for a SCL init.
- Can you find HDIs responsible for it? One or many HDIs in a row showing high DVDD current (look in the SMT DVDD CURRENTS GUI) can stall the readout. Ask for a run pause and redownload the problematic HDIs from the corresponding SEQ in the download GUI (SMT Download GUI).  Note!:  If the problem HDI is tripped, remember to ramp down and turn off the HV first.
- Ask for a run pause and do a "Reinit VME" for that crate in the SMT Download GUI.
- Ask for a run pause and reinit the sequencer controller for that crate in the SMT Download GUI (left click on the SEQC_**** box near the top of the crate’s details window, then click on "reset" and "init")
- Ask for a run pause and do a "download" for the whole crate (SMT Download GUI)
- If all of the above fails: Ask for a run pause, power cycle the corresponding VME crate, do a "reinit VME », once the Power PC has rebooted, and continue the run. How to powercycle a VME (=VRB) crate?  go to 2nd floor Movable Counting house; in the upper part of each rack, there are two 3 pull switches labeled with the crate numbers present in that rack. Flip the corresponding switch down to turn the crate off, wait a few seconds and flip the switch back up. It takes 30 seconds to a minute for the crate to reboot.
- Page the SMT expert (218-8764)

## Shift Guidelines

- HV trips when ramping up :See the « Troubleshooting HV pod Trips » section of the « High Voltage » chapter of the SMT Shift Instructions.
- Check for major and minor HV alarms:
- If they are related to currents which are getting close to the trip limit, increase the limit by 10uA using the HV current limit GUI(GUI starter - page B - Button 4). If you need to do it more than 3 times in your shift for the same pod, page the expert (218-8764).
- If they are related to voltages, check that you ramped the voltages up to 100%.
- if not, fix it.
- if so, page the SMT expert (218-8764)
- Check for major « Occupancy » or « Dead » alarms. The ONLY monitoring alarms that you can disregard are the « Occupancy » alarms, ONLY when the SMT HV is OFF. In ALL other cases, IF the monitoring is running, you should take the « Dead » and « Occupancy » alarms seriously. Try to fix the alarms by redownloading the corresponding HDIs. You may need to wait 5 to 10 munites to see the effect of your doanload (check the « SES UPDATE TIME » for the corresponding crate - see the « Monitoring » part of these Guidelines for more details).If the problem persists, page the expert (218-8764)
- Check smt_examine histograms and compare them with the references. Write in the logbook any discrepancy you can see.
- At the end of your shift, fill in the « Shift Summary » check list and logout from the logbook.

## Types of HDIs present in D0SMT

- Barrels: 1 HDI = 1 Ladder
- 3 chips: Bi-j-k k (i=1,6; j=1,2,5,6) 2 Single Sided sensors
- 6 chips [3 p-side + 3 n-side]: Bi-j-k k, (i=2,3,4,5; j=1,2,5,6)1 Double Sided 90o stereo sensor
- 9 chips [5 p-side + 4 n-side]: Bi-j-k i, k, (j=3,4,7,8) 2 Double Sided 2o stereo sensors
-           where Bi-j-k means Barrel i, Layer j, Position k
- F-Disks: 1 wedge = 1 Double Sided sensor               1 HDI    = 1 side of a wedge
- p-side: 8 chipsFi-1-k i, k
- n-side: 6 chipsFi-2-k i, k
-           where Fi-j-k means Disk i, Side j, Position k
- H-Disks: 1 HDI = 1 side of a wedge
- p-side: 6 chipsHi-1-k i, k 2 Single Sided sensors
- n-side: 6 chipsHi-2-k i, k2 Single Sided sensors (it is actually a second p-side)
-           where Hi-j-k means Disk i, Side j, Position k

## Correpondence between different mappings

| SMT Download GUI | Smt_Examine | Online Monitoring and Offline |
| --- | --- | --- |
| B2-L5-HDI09-(N) | B2-5-09-p | B2-5-9 |
|  | B2-5-09-n |  |
| H4-L1-HDI03-(N) | HD 4 03-p | H4-1-3 |
| H4-L2-HDI03-(N) | HD 4 03-n | H4-2-3 |
| F3-L1-HDI05-type8-(N) | FD 3 05-p | F3-1-5 |
| F3-L2-HDI05-type6-(N) | FD 3 05-n | F3-2-5 |

- « L » identifies:
-  the layer (1,...,8) in a barrel
-  the side (1,2) of a disk

## D0SMT

- detector types

- L3

- L6

- L6

- L9

- L9

- F6

- F8

- H6

- n-side

- n-side

- n-side

- p-side

- p-side

- p-side

## SE

- B5

- B4

- B6

- B2

- B3

- B1

- NE1

- NE0

- NW0

- NW1

- SW1

- NW

- NE

- SW

- INTERFACE
- BOARD
- CRATES

- ADAPTER
- CARDS

- 80/3M
- CABLE

- L.M.
- CABLE

- SE0

- 120 ladders
- 72 F-wedges
- 48 H-wedges

- SE1

- SW0

- H2
- H1

- H4
- H3

- Central
- Calo

- CFT

- F6
- F5
- F4
- F3
- F2
- F1

- F12
- F11
- F10
- F9
- F8
- F7

- Calo North End Cap

- Calo South End Cap

- Muon wall

- Muon wall

- Muon wall

- Muon wall

- East Cathedral

- West Cathedral

## Cathedral

**~19’-30’ High **
**Mass Cable     **
**(3M/80 conductor)**

**1553 **
**Monitoring**

**Serial Command Link**

**Cathedral**

**Interface**
** ****Board**
**Crates**
**(8x18)**

**8’ Low Mass Cables**

**3/6/8/9 Chip HDI**

**Sensor**

**CLKs**

**Horse Shoe**

**3/6/8/9 Chip HDI**

**Sensor**

**Adaptor **
**card**

**25’ High Mass Cable (3M/50 conductor)**

**CLKs**

- I
- B

**MCH2**

**V****R****B****C**

**S**
**B**
**C**

**V****R****B**

**Monitoring**
**SDAQ**

**PDAQ  (L3)**

**VRB**
**Crates**
**(12x10)**

**Platform**

**SEQ**

**SEQ**
**Controller**

**Optical Link**
**1Gb/s **

**Sequencer **
**Crates**
** ****(6x20)**

**VTM**

**Around Iteraction Region**

**CLKs**

**MCH3**

**PowerPCs and Single Board Computers are accessed thru Ethernet**

**MCH2**

**HV **
**Crates**
** ****(8x6+2x4)**

**Cathedral**

**Fuse**
**Panel**

**Bulk LV**
**Power Supplies**

**HV**
**fanout**
**1=>4**

**HV**
**breakout**
**box**

**VME**
**Crates**
**(4x3)**

**25 twisted **
**pair cables**

**17 twisted **
**pair cables**

**HV**

**LV**

**I,V,T Monitoring**

## About LV and HV (1)

- Our silicon detectors are made of 1 or 2 silicon sensor(s) glued to an HDI which carries
- SVXIIe chips. These chips allow to read out information about the charge collected on the
- sensor strips, when charged particles went through the sensor(s). For the detector to
- function properly:
- the silicon sensors must be depleted. This requires 1 (single sided sensors) or 2 (double-sided sensors) HV power supplies.
- the SVXIIe read out chips must be powered. This requires 3 different LV power supplies: AVDD, AVDD2, and DVDD.
- HV and LV are provided by the IB to the HDI which, in turn, provides it to the sensor(s) and
- to the SVXIIe chips, respectively.

## About LV and HV (2)

- In the SMT download GUI, when you click « download », e.g. in the Left-click menu of a
- Sequencer or a Sequencer Fiber, what happens for the corresponding HDIs is the following:
- The Sequencer asserts the HDI enable line for the corresponding IB channel:
- if the SVX chips on the HDI are already powered, nothing new happens.
- if the SVX chips on the HDI are not powered:
- if there is a trip condition, i.e. HDI AVDD, AVDD2 or DVDD current greater than 700mA, or HDI temperature greater than 40oC, the IB channel will be kept OFF and a corresponding Trip Bit(s) will be asserted, making the corresponding IB channel turn red in the SMT download GUI and blink red/black on the DVDD current GUI. Resetting the trip condition requires to do, in the SMT download GUI, a « HDI off » in the Left-click menu of the tripped HDI. This will, of course, work only if the cause of the trip has disappeared before you try to reset the trip condition.
- if there is no trip condition, the firmware in the FPGA of the IB will try to turn, in sequence, the IB switches for AVDD, AVDD2 and DVDD:
- if none of the currents for these 3 HDI supplies is greater than 700mA and the HDI temperature is not greater than 40oC, the IB buffer for the HDI data lines will be enabled and the IB HV switches will be turned on.
- if not, all switches will be turned off, and the corresponding Trip Bit(s) will be asserted, making the corresponding IB channel turn red in the SMT download GUI and blink red/black on the DVDD current GUI (see above).
- Download the chips of the corresponding HDI with the initialization parameters as defined in the « SVX parameter » item of the « Global Parameter » button  Left-click menu in the SMT Download GUI
- When you click on « HDI off » , e.g. in the Left-click menu of a Sequencer, a Sequencer
- Fiber or an HDI, the Sequencer de-asserts the corresponding HDI enable line(s), which makes
- the FPGA in the corresponding IB channel(s) disable the data buffer and turn off the switches
- for DVDD, AVDD2, AVDD, +HV and -HV. This will not turn off the HV pods,
- it will only turn off the IB switches connected to the HV HDI lines.

- General rule for HDI(s): 	Turn ON LV first, then HV.
- 			Turn OFF HV first, then LV.

## SEQuencer

- 50 cond

- 34 cond (8HVs + 1 temp)

- (platform)

- Optical
- Fiber

- 1

- 2

- 3

- 0

- VTM
- (MCH2)

- VRB
- (MCH2)

- FusePanel
- (cathedral)

- Power
- Supply
- (cathedral)

- Power
- Supply
- (platform)

- RMI/RM
- (cathedral)

- RMI/RM
- (platform)

- AVDD
- AVDD2
- DVDD
- VCC
- +15V

- SHV
- cable

- 50 cond

- +5V
- -5V

- SMT cooling system

- Coolant Flow

- Coolant IN/OUT Temp

- Temps of selected
-          devices

- Cryo
- Permit

- Drip

- Water Flow

- Hygrometry

- Smoke (HSSO)

- Control/
- Status

- Control/
- Status

- 24 SHV inputs
- HV Fan Out box
- (MCH2)

- HV Break Out box
- (platform)

- HV
- Power
- Supply
- (MCH2)

- 12 cond

- Air Flow

- Water Flow

- Drip

- Smoke

- 0-1

- 2-3

- 4-5

- 6-7

- 0

- 7

- Adaptor
- Card
- (Horse Shoe)

- 80 cond

- Low Mass Cables

- Chain B

- Chain A

- Ladder

- Ladder

- CLK/CLKB

- CLK/CLKB

- monitoring
- chan. A:
- AVDD
- AVDD2
- DVDD
- Temp

- hdi A
- enable

- trip A

- +HVA

- -HVA

- dvdd

- avdd2

- avdd

- ON/OFF

- FPGA

- buffer A enable

- data/control lines

- 0
- 1
- 2
- 3
- 4
- 5
- 6
- 7

- Interface Board
- (cathedral)
- details given for chan. A only
- same for other 7 channels

## About LV and HV (4)

- Now, up to 4 HDIs can share the same HV pod (single-sided detector) or the same pair of
- HV pods (double-sided detectors). For instance, the 9-chip ladders (double-sided) B5-7-4,
- B5-7-5, B5-8-5, B5-8-6 share positive pod 200P and negative pod 201N:

- If the Download GUI would let you download B5-8-6 with pods 200P and 201N on, it
- would make the corresponding Sequencer assert the HDI enable line for B5-8-6, making
- the LV and HV switches turn ON on the corresponding IB channel. The problem here is
- that as soon as the switches are on, since the input of the SVX chips are at a virtual ground
- when powered, the coupling capicitors on the n-side(resp. p-side) of the silicon sensors of
- B5-8-6 will see a volatge step of +HV (resp. -HV). These capacitors are very delicate, to
- avoid stressing them and breaking them prematurely, it’s better to slowly ramp the HV on
- them.
- So, to make a long story short, to make things safe, each time you need to turn ON or OFF
- the LV on an HDI, first, you ABSOLUTELY need to ramp down and turn OFF the HV on
- that HDI and on all the HDIs which share the sequencer with the HDI you want to deal with
- (how to do it is described on page 22). The download GUI will not execute your command
- (and print an error message on the terminal window) if you do not comply.
- If you want to redownload HDIs which already have their LV ON, you can do it without
- touching the HV. The LV will not be touched, but the SVX chips of the corresponding
- HDIs will be re-initialized with the current download parameters.

## SVXIIe read out chips download parameters

- you should redownload all the crates with these default parameters at the beginning of each shot setup, to make sure the SVX chips are properly initialized for Physics!

## High DVDD currents and HDI LV Trips (1)

- If a lot of HDIs show high currents in the DVDD Current GUI, that usually means that the SVX chips are not read out. This happens when the crates are not in a global run, or when the global run is pause or stalled for some reason. **You should not let the system in that state for too long**.

- DVDD Currents GUI color coding:  orange:	LV is OFF  green: 	LV is ON and I_DVDD OK  purple:	LV is ON and I_DVDD slightly high  red:	LV is ON and I_DVDD high  red/black:	Abnormal state (Trip or Buffer Disabled)
-   light blue:	Channel is a disabled 6-chip F-wedge used
- 	only to provide bias to the 8-chip partner
-   pink:	Channel permanently disabled
-   grey: 	Channel not used

## High DVDD currents and HDI LV Trips (2)

- A way to lower the DVDD current drawn by the SVX chip, is to send trigger to the Sequencer Controllers so as to make the chips work. This can be achieved in several ways, by:
-  having the crates in a working Global Run,
-  having the crates out of the Global Run and making use of a special run which just sends triggers to the sequencer crates (the VRB crates are not read out). This is described in the « SMT Crater » section,
-  sending triggers to the Sequencer Controllers via the 1553, by pressing the « turn pulses ON » button in the SMT Download GUI main page. Do not forget to « turn the pulses OFF » before the Global Run starts or resumes readout of the SMT crates. If you try to read out the crates while pulses are ON, this will result in 100% L1 FEB, because you are sending more triggers to the sequencer crates than to the VRB crates.

## High DVDD currents and HDI LV Trips (3)

- If some HDIs consistently draw too much current (red or purple on the DVDD Currents GUI)
- you need to REDOWNLOAD them. The first step is to find what VRB they are connected to.
- This can be done looking at the IB mapping chart:

## High DVDD currents and HDI LV Trips (4)

- A Left-click on an item of a menu executes that item
- A Right-click on a menu closes it.

**High DVDD currents and HDI LV Trips (4)**

- Then, go to the corresponding VRB in the SMT Download GUI, and click download on the Left-click menu of the corresponding sequencer.

- If the download was successful, the HDIs should appear green, both in the Download GUI and on the DVDD Currents GUI:

## High DVDD currents and HDI LV Trips (5)

- If one or more HDIs appear red in the Download GUI, the download was not successful.

- Try to download a few more times at the Sequencer level. If it still does not work, try to download at the corresponding Sequencer Fiber level.

-  HDI Status Color Coding:
-  white: permanently disabled
-  yellow: disabled
-  grey: enabled and either          OFF or unknown
-  green: enabled, ON and            downloaded OK
-  red: enabled,         (ON and failed download)        or tripped

- IB Channel Status
- Color Coding:
-  green: OK
-  red: tripped

- Interface Board

- Sequencer Fibers

- Sequencer

- VRB Fibers

- VRB

- 1 Fiber = 2 HDIs

## High DVDD currents and HDI LV Trips (6)

- 1. If this still does not work, it sometimes helps, when an **HDI is reluctant in downloading properly**, to turn it OFF and ON again.
- 2. In the same way, if an **HDI has tripped**, to recover it, you need to clear the trip by turning the HDI OFF and then turn it back ON again, by downloading it.

- TRIP

**In both cases, if the High Voltage is ON, you will need first, to RAMP DOWN and TURN OFF the HV on ALL HDIs connected to the SAME SEQUENCER as the HDI you want to turn ON or OFF. Otherwise, the Download GUI will not execute your commands.**

## High DVDD currents and HDI LV Trips (7)

- In the left-click menu of each Interface Board connected to the sequencer, the corresponding HDI(s) is(are) connected to, click on « start HV GUI  ». For each IB a GUI will pop up showing the pods you need to ramp down and turn off. In our example, HDI B3-L4-HDI04 is connected to sequencer SEQ_030B14. That sequencer is connected to 2 IBs, namely: INT_NW_0B0E and INT_NW_0B10. Start the HV GUI for the first IB:

- Do the same for the second IB:

- In all the HV GUIs, do « Unlock », « Set HV to 0% », « Ramp », and when the all the « V_Read » values are below 3V, click on « Off ». You are now ready to take care of the problematic HDI.

## High DVDD currents and HDI LV Trips (8)

**B3-L4-HDI04-(N)**

- OFF

- Turn OFF the tripped HDI:

- As explained earlier, try to download the HDI at the sequencer level or at the Sequencer Fiber level.
- If the download worked, in the HV GUIs shown in the previous page, turn the HV ON, « Set HV->100% », « Ramp ». Once the target voltages are reached, « Lock », « File->Quit », and you are done!
- If not, you need to turn the HDI off, disable it as explained in the next page ...

## High DVDD currents and HDI LV Trips (8)

_**Disabling Instructions**_

## High DVDD currents and HDI LV Trips (9)

## HV current limit GUI

- write HV pod
- name here

- click here to get
- the current limits

- Minor Alarm
- Current Limit

- Major Alarm
- Current Limit

- Current Trip
- Limit

- Actual Pod
- Read Back Current

- click here to increase the
-  3 current limits by 10uA

- current limits are updated and
- a mail is sent to Breese and Eric

- click here to
- quit the GUI

- Start the HV current limit GUI using the « GUI starter » GUI, Page B, button 4
- (to start the GUI starter, type in a terminal window: cd; setup d0online; ./startguis.py)

## Monitoring (1)

- Monitoring of the data quality is done at the strip level in processes which run in the VRB crates Power PCs (IOCs). The status of these IOCs can be looked at from the « Readout IOC Monitor » (from GUI starter, page C, button 6).

- Power PCs (IOCs)
- in the 12 VRB crates

- CPU consumption varies between 50 and 75% when monitoring is running

- For each VRB crate, the monitoring process captures events from the SBC and makes histograms out of them. A Linux box queries the IOCs one after the other to get their histograms out, so as to allow their browsing through a WEB interface.  When an IOC is queried, if the number of events accumulated by the IOC lies between 8,000 and 10,000, an update will be sent for the corresponding crate to the Significant Event System (the result of which is shown in the Alarm Display) for major alarms related to the « Occupancy » and « Dead » categories. An « Occupancy » alarm is set whenever the occupancy of an HDI is greater than 25%. A « Dead » alarm is set if (a) chip(s) of an HDI never appear in the read out stream. Note that disabled HDIs do not generate a « Dead » alarm.  You have to take Occupancy and Dead alarms seriously!

## Monitoring (2)

- You can connect with Netscape (or you favorite Browser) to the Monitoring Server page
- _[(http://www-d0ol/smtMonitoring/)](http://www-d0ol/smtMonitoring/)_. It also appears labeled as « SMT Monitoring Page » in the Netscape Toolbar and in the Bookmarks menu.

- click on this to update the page

- VRB crates

- Last time data were received from IOC in VRB crate 62

- Last time update was sent to SES (Alarm system) for crate 65

- As a general rule, the monitoring should always be turned on during Global Runs (zero_bias or physics). Check this by looking at the IOC CPU consumption, on theReadout  IOC Monitor GUI, at the « IOC DATA UPDATE TIME » and « SES UPDATE TIME » on the Monitoring WEB page. The monitoring is started at the beginning of a run and stopped at the end of a run. If monitoring does not seem to be on, during a Global Run, check with the DAQ shifter that the trigger file has SMT monitoring enabled. If not, the DAQ shifter should enable the monitoring and start a new run. You can also start the monitoring « by hand », VRB crate by VRB crate, usingthe crate « details » page in  the SMT download GUI, as follows:

## Monitoring (3)

- Imagine that for some reason, you need to power cycle crate 66  during a global run. After pausing the run, you cycle the power on crate 66, you wait for the Power PC to reboot, you « Reinit VME » and you resume the run. Now, what you see on the IOC monitoring is that the monitoring process in crate 66 is probably not processing anything:

- open the details page for crate 66 in the SMT download GUI and click on the « Start monit run » item in Left-clickmenu of the VRBCR_66 button. In the resulting pop up window, write the current run number and click « Start »

- You can open a terminal window on the IOC by connecting to it through the MCH2 terminal server. The login name is: ioc
- the passwd: 0nlined0

## Monitoring (4)

- If everythig goes as expected, this is what you should then see:

- After 5 to 10 minutes, you can also check that the « IOC DATA UPDATE TIME » and « SES UPDATE TIME » are current on the Monitoring WEB interface.

## LogbookStarting

- The very first thing that a shifter should do is log into the online electronic
- logbook.  Normally it is already running on one of the SMT monitors.  If it is
- not, then start the logbook:
- > setup d0online
- 	> start_daq logbook &
- 	Or from the "startgui" gui D.1.
- Click on the **SMT** tab.
- Select **SMT Log **from the **Log** menu.
- An SMT Log window should now appear in the central field, with pre-
- configured checklist icons to the right:

## LogbookUse

_**General Guidelines**_: *Everything* that is done to any part of the SMT system by the shifter or anyone else must be recorded in the logbook.  Some guidelines to follow are:
**Use Checklists**: The pre-configured checklists are there to provide a thorough framework and guide for conducting a shift.  They are required, but not sufficient; i.e. a starting point for logbook entries.
**Keep it short**: Each entry should be concise and focussed on a single task/observation/event.  Practice a “log-as-you-go” habit instead of saving up a lot of information for a big summary entry.
**Archive promptly**: Archive entries immediately  upon *timely *completion.  Do not keep entries open for extended periods.  For example, do not wait for a Shot Setup to complete before archiving the Shot Setup checklist.  Archive promptly, and annotate later, if necessary.  Also, do not make logs within entries (e.g. 8:00 – started shift.  8:30 – ramped up HV.  9:00 – recovered all dead HDI’s).
_**Log In/Log Out**_: To log into the logbook, click on the **Log In/Log Out** button at the top left.  Enter your account name and password in the window that pops up (to obtain an account, contact Harald Fox or Eric Kajfasz).  Remember to log out of the logbook at the end of your shift.
_**Checklists**_: There are twelve pre-configured checklist icons to the right.  To use one of the checklists, select an icon and drag it into the SMT Log window.  Then proceed to complete the checklist by filling in all of the fields.
- 	The checklists are:
**Begin Shift**: complete at the beginning of a shift
**Between Stores**: fill out (and update once per hour) during the time between the last store and the next shot setup.  This includes during beam studies.
**Quiet Time Calibration**: complete any time a calibration run is taken.
**Shot Setup**: complete when MCR is preparing for shot setup.
**Begin Store**: complete after a new store has been verified with the Shift Captain.
**Begin Run**: fill out any time a new data run is started.
**Data Taking**: complete a new Data Taking checklist once every hour during an ongoing data run.
**End Run**: complete at the end of every data run.

## LogbookUse

**End Store**: complete before a store is dumped (or after a store is lost unexpectedly).
**Shift Summary**: complete at the end of a shift (this summary will be automatically e-mailed to the d0smt mailing list).
**HDI TROUBLE**: fill out any time an individual HDI erquires special attention (i.e. redownload due to FEB or high currents,  disable, re-enable, etc.).
**CONFIG Change**: fill out any time the SMT run configuration has changed (i.e. any time an HDI has been disabled/re-enabled or there has been a change in the voltage setting of any of the HV pods).
_**Text Entry**_: To add a text entry to the logbook, select the **Text** icon from the right and drag it into the SMT Log window.  Then compose in the generated entry.
_**Creating Images**_: Many times it is required or desired to make an image of a plot or gui window on one of the monitors.  This can be done with the **import** command:
- Open a shell on the screen in which the desired image (plot, gui, etc.) appears.
**cd** to the appropriate SHIFT directory which may need to be created (e.g. ~d0smt/SHIFTS/2002/oct/16/day).
- >import <filename>
- Place the ‘+’ cursor on the desired window to be imaged, and left-click.  The image will be created as <filename> in the current directory.
_**Inserting Images**_: There are two ways to insert images into the logbook:
**GIF/JPG Entry**: Select either the **GIF** or** JPG** icon from the right and drag it into the SMT Log window.  A **Get File** pop-up window will appear from which you can browse for the desired image.

## LogbookUse

**Insert into an existing entry**: with the cursor positioned within an existing logbook entry, select **Image > From a File** from the **Insert** menu on the SMT Log toolbar. A **Get File** pop-up window will appear from which you can browse for the desired image.
_**Archiving Entries**_: Any entry to the logbook should be archived immediately upon completion.  To do this, left-click on the entry header, then right-click and select **Archive Selected Entries**.
_**Annotating Entries**_: An annotation can be added to any archived entry by clicking on the **ANNOTATE** button in the archived entry.

## DØSMT Operations Database

- The Operations Database is a web-based database that we use to keep track of
- the performance history of the SMT.  By far, the most-used section of the
- Operations Database is the  one in which we keep track of individual HDI
- histories.  In particular, it is a shifter’s responsibility to make a database entry
- each time that an HDI is disabled or re-enabled.
- The Operations Database can easily be located from a netscape window started
- from the d0smt account.  A link to “D0SMT Operations Database” can be found
- at the far right of the quick link bar.
- If you need to make an entry into the database for a particular HDI (e.g. when an
- HDI has been disabled or re-enabled) , go to the “IB Testing” section in the left-
- hand frame. Let’s assume you have disabled SW1B11-B:
- From the pull-down menus in the “IB Testing” area, select: “Edit” “SW” “1”. Then click “Go for it!”.

## DØSMT Operations Database

- You will be prompted for a user name (d0smt) and password (dab).  Enter those at the appropriate spaces and click “Submit”.
- A graphic map of the SW interface board crates  will appear.  Click on the colored box corresponding to the IB channel for the HDI that you want to make an entry for.                                                                                **NOTE!:** In the download GUI and elsewhere, when HDIs are referred to by their Interface Board channel, the numbering is in hexadecimal (e.g. the ’11’ in SW1B11-B).  In the Operations Database, two designation are given: the top ‘IB #’ row is in hexadecimal, the bottom ‘GUI #’ row is in decimal.  Be careful that you are selecting the correct HDI!

## DØSMT Operations Database

- A list of all entries made for that particular HDI will appear.  To make a new entry, click on  the “New Rec” button near the upper left.
- A new record template will appear.  Fill in your initials, choose a “Status” representing the current state of the HDI, and enter some thorough comments describing the problems observed, what actions were taken and why.  The fields for individual currents do not, in general, need to be filled in.

## DØSMT Operations Database

- When the entry has been completed, either click “Update”, which will return you to the HDI’s history (where your new entry will appear), or click “Update/Top”, which will return you to the interface crate map.

## All-GUIs-starter

- This GUI has been written to make your life, as a shifter, easier. It will allow you
- to start most of the other GUIs you will need to perform your work.
- To start this GUI:
-  open a d0smt shell window
-  type:
- > setup d0online
- > cd ~or
- > cd ~/monitoring  (it doesn not matter, as you whish...)
- > ./startguis.py &

- Page A: monitoring GUIs related to Low Voltages
- and temperatures
- Page B: monitoring GUIs related to High Voltage
- Page C: monitoring GUIs related to download,
- calibration and data taking
- Page D: miscellaneous GUIs

## All-GUIs-starter General Monitoring GUIs 

- The General Monitoring GUIs are the ones which allow you to monitor the behavior of the detector as far as:
-  HDIs DVDD currents
-  Interface Board Low Voltage
-  Interface Board Temperature
-  High Voltage.
- They need to be active at ALL TIMES. Using the all-GUIs-starter, they can be started individually (buttons B.1., A.2., A.3., A.4.) or all at once (button A.1.)

## HDIs DVDD current GUI

- this GUI displays the DVDD currents of each Interface Board Channel ( i.e. each HDI)
- It can be started from the all-GUIs-starter (A.2.) or by typing in a d0smt shell window:> setup d0online> cd ~/monitoring> ./IBidvdd.py &

- NW

- NE

- SW

- SE

- 0

- 1

- 0

- 1

- DVDD Currents GUI color coding:  orange:	LV is OFF  green: 	LV is ON and I_DVDD OK  purple:	LV is ON and I_DVDD slightly high  red:	LV is ON and I_DVDD high  red/black:	Abnormal state (Trip or Buffer Disabled)
-   light blue:	Channel is a disabled 6-chip F-wedge used
- 	only to provide bias to the 8-chip partner
-   pink:	Channel permanently disabled
-   grey: 	Channel not used

## HDIs DVDD current GUI

- B5

- B4

- B6

- B2

- B3

- B1

- SW1

- SW0

- NW0

- NW1

- SW1

- SE

- NW

- NE

- SW

- INTERFACE
- BOARD
- CRATES

- ADAPTER
- CARDS

- 80/3M
- CABLE

- L.M.
- CABLE

- SE0

- 120 ladders
- 72 F-wedges
- 48 H-wedges

- SE1

- SW0

- H2
- H1

- H4
- H3

- Central
- Calo

- CFT

- F6
- F5
- F4
- F3
- F2
- F1

- F12
- F11
- F10
- F9
- F8
- F7

- Calo North End Cap

- Calo South End Cap

- Muon wall

- Muon wall

- Muon wall

- Muon wall

- East Cathedral

- West Cathedral

**Location of the Interface Board Crates in the Cathedral**

## HDIs DVDD current GUI

- The IB crates are subdivided into 2 halves (A and B), each half corresponds to 9 IB boards. In the IB crate GUI each IB is represented by a row:
-  The first cell of the row contains the name of the interface board.    The name is constructed as follows:
-  quadrant (NE, NW, SE, SW)
-  crate number (0, 1)
-  crate half (A, B)
-  IB slot number (in hex: 02-0A and 0C-14)
-  It is followed by 8 cells corresponding to the 8 IB channels  (labeled A thru H). Each IB channel can be connected to an HDI.    The numbers written in the cells give the DVDD current in mA drawn by   the corresponding HDIs
-  if a cell is gray and has a number, it means the    HDI is really dead and thus never enabled.
-  if a cell is gray without number, it means that the    IB channel is not used (no HDI connected to it)

- 1 row = 1 Interface Board

- IB channel
- not used

- half-crate A

- half-crate B

- dead HDI
- never enabled

- HDI drawing
- 106 mA on DVDD

- IB name

- A  B  C  D  E  F  G  H

- IB channel labels

## HDIs DVDD current GUI

- if you click here

- if you click here

- IB Channels

- HDI currents

- HDI voltages

- HDI
- temperatures

- IB Channel
- detailed status

- IB Channel
- status

- click

- IB channel name
- corresponding
- to that cell

- HDI (offline) name
- corresponding
- to that cell

- HV pod(s)
- connected
- to that cell

- names of the all
- the HDIs sharing
- these HV pod(s)

- sequencer
- corresponding
- to that cell

- VRB
- corresponding
- to that cell

## HDIs DVDD current GUI

-  Check that the GUI is updating (e.g. from the GUI title bar date/time) if NOT kill it and restart it
-  You should only see a small number of HDIs with a purple or red color. Some of the causes for this to happen can be:
-  if only 2 to 4 HDIs become purple or red in a row (i.e. Interface Board), this usually means the corresponding HDIs need to be redownloaded. To figure out what VRB they correspond to, you can either click on the corresponding cell in the DVDD current GUI or look at the IB/VRB map posted close to the SMT monitors
-  if HDIs are purple and red everywhere, it may mean that the HDIs are not read out:
-  check with the DAQ shifter that all the SMT readout crates are in a global run
-  if they are not, ask him/her to include them
-  if for some reason the global DAQ has to run without the SMT crates:
-  if the DAQ shifters allows it, start a SEQ only run with your own **taker**, after having prepared the SEQ only trigger file with the **crater** GUI
-  if the DAQ is down or the DAQ shifter does not want you to start your own taker, **turn on the pulsers** from the main page of the download GUI. DO NOT FORGET to turn OFF the pulser whenever the SMT crates are put back in a global run.

## IB Power Supplies GUI

- There are 8 sets of IB Power Supplies. One set for each IB crate. Each set of power supplies is comprised of: 1 15V PS for IB 1553 circuitry, 3 VCC PS to power the IB ICs (1 for the 1st 7 boards, 1 for the next 6 boards, and 1 for the last 5 boards), and 6 PS to supply power to the SVX chips (1 AVDD2 PS, 2 DVDD PS [1 for 3 chippers and 1 for 6,8 and 9-chippers], 3 AVDD PS [1 for 3 chippers, 1 for 6 chippers and 1 for 8 and 9 chippers])
- To start the IB Power supply GUIs, open a d0smt shell window:
- 	> setup d0online	> cd ~/monitoring	> ./IBLVsmall.py &
- or
- 	 in the all-GUIs-starter click button 3 on page A

- detailed
- digital status

- OR-ed Status

- click for details

- The External Interlock can be activated by the:
-  Cryo permit (HDI cooling)
-  IB Rack Monitors (Flow, Leak, Smoke)
-  MCH3 Switches to AC power of the IB PS

## SEQ Power Supplies GUI

- To start the Sequencer Power supply GUI, open a d0smt shell window:
- 	> setup d0online	> cd ~/monitoring	> ./seq_gui.py &
- or
- 	 in the all-GUIs-starter click button 6 on page A

- power supplies
- for the 3 North
- sequencer crates

- power supplies
- for the 3 South
- sequencer crates

- powers off **ALL** North crates
- (3 SMT and 1 CFT)

## IB Temperature GUI

- All IB crates have a set of 3 pairs of temperature sensors inserted beween some of their IBs. The sensors are read out through IB channels which are not used for HDIs.
- If 3 or more temperatures out of 6 in a crate are over 35 degC, we consider that there is problably a cooling problem on that crate. In that case the GUI will automatically switch off the LV power supplies of that crate. So it is really IMPERATIVE to check from time to time that this gui is updating.
- To start the IB Temperature GUI, open a d0smt shell window:
- 	> setup d0online	> cd ~/monitoring	> ./IBdegC.py &
- or
- 	 in the all-GUIs-starter click button 4 on page A

- click for
- details

- 3 pairs of temp
- sensors per crate

- check that
- time changes

## Utility Scripts and GUIs

- To prepare to run the utility GUIs:, open a d0smt shell window:
- 	> setup d0online
- 	> cd ~/monitoring
- Since some of these utility GUIs can be launched from the all-GUIs-starter, you may want to start it too:
- 	> ./startguis.py &
- Some of the most useful utility GUIs are:
-  information about what VRB, IB, SEQ, HV pod(s) a specific HDI (e.g. B2-3-5) is connected to:	> ./show_det_info.py B2-3-5 &
-  information about what is connected to a specific HV pod (e.g. 100P):	> ./show_hv_info.py 100 &(rem: you must drop the P or N from the end of the pod name)
-  information about a specific IB channel (e.g. SW0A02-C)	> ./show_IB_info.py SW0A02-C &
-  information about specific fiber input (e.g. fiber 2 [possible choices: 1, 2, 3 or 4]) of a specific VRB (e.g. VRB 13 in crate 6B)	> ./show_VRB_info.py VRB_6B13
-  information about a specific VRB channel (e.g. channel 2 [possible choices: 0 thru 7]) of a specific VRB (e.g. VRB 13 in crate 6B):	> ./show_VRBchan_info VRB_6B13 2

## Utility Scripts and GUIs

-  script to check **fast** and **slow** archivers:	> cd ~	> archiverchecks.sh &or	 in the all-GUIs-starter click button 7 on page D

- there should be
- 3 processes

- there should be
- 3 processes

- date and time
-  should be current

- date and time
-  should be current

- If this is not the cas, follow the instructions in the « Archiver » section to stop and
- restart the problematic archiver(s)

## Listing Scripts and GUIs

-  list of HDIs which are turned off	> ./listDisabledHDIs.py &or	> ./show_disabled_hdis.py &or	in the all-GUIs-starter click button 4 on page D

- script style

- GUI style

## Listing Scripts and GUIs

-  list information about HV	> ./list_underdepleted_hdi.py &or	> ./show_underdepleted_hdis.py &or	in the all-GUIs-starter click button 5 on page D

- script style

- GUI style

## Rack Monitor Interface GUI

- To start the Rack Monitor Interface GUI, open a d0smt shell window:
- 	> setup d0online	> cd /online/config/ctl	> ./pfm.rmi &
- or in the all-GUIs-starter click button 7 on page A

- SEQs
- North

- SEQs
- South

- Central Platform RMs
- Sequencer Crates

- Cathedral RMs
- Interface Board Crates

## Read Out Crates IOCs

- To run the Read Out Crate IOC GUI, open a d0smt shell window:
- 	> setup d0online	> cd ~/onl_smtnew/util	> ./smtreadout.ioc &
- or
- 	 in the all-GUIs-starter click button 6 on page C

- click to reboot
- a particular IOC

- click to reboot
- all IOCs

- when trying to reboot
- you will be asked for
- a password. It is:
- 0nlined0

- terminal server port
- to which the IOC
-  is connected

## Read Out Crates IOCs

- To connect to a particular IOC (e.g. IOC of crate 65 i.e. port 17):

- list process in the IOC

- bad state for a
- process to be in

- 0nlined0

## 1553 Controller Crates IOC mapping (MCH3 Rack 300 terminal server t-d0-mch3)

| Quad | IOC | 1553 Port | Devices Connected | Cable info | Termnl
Port |
| --- | --- | --- | --- | --- | --- |
| SW | d0olctl57 | 0 | 10 SEQ PC19-0A - Sequencers | black #1 | 34 |
|  |  | 1 | 10 SEQ PC19-0B - Sequencers | black #2 |  |
|  |  | 2 | 10 SEQ PC20-1B -Sequencers | black #8 |  |
|  |  | 3 | 3 South SEQC – Seq. Controllers (South) | black #9 |  |
|  |  | 4 | 18 IB SW-0 | black #17 (G) |  |
|  |  | 5 | 18 IB SW-1 | black #18 (H) |  |
| SE | d0olctl56 | 0 | 10 SEQ PC20-0A - Sequencers | black #5 | 1 |
|  |  | 1 | 10 SEQ PC20-0B -Sequencers | black #6 |  |
|  |  | 2 | 10 SEQ PC20-1A – Sequencers | black #7 |  |
|  |  | 3 | 4 East RMI (IB power supplies/East) | black #19 (A) |  |
|  |  | 4 | 18 IB SE-0 | black #20 (C) |  |
|  |  | 5 | 18 IB SE-1 | black #21 (D) |  |
| NW | d0olctl60 | 0 | 10 SEQ PC03-0A - Sequencers | black #10 | 36 |
|  |  | 1 | 10 SEQ PC03-0B - Sequencers | black #11 |  |
|  |  | 2 | 10 SEQ PC03-1A -Sequencers | black #12 |  |
|  |  | 3 | 4 West RMI (IB power supplies /West) | black (green H) |  |
|  |  | 4 | 18 IB NW-0 | black #22 (I) |  |
|  |  | 5 | 18 IB NW-1 | black #23 (J) |  |
| NE | d0olctl59 | 0 | 10 SEQ PC03-1B - Sequencers | black #14 | 6 |
|  |  | 1 | 10 SEQ PC04-0A - Sequencers | black #15 |  |
|  |  | 2 | 10 SEQ PC04-0B - Sequencers | black #16 |  |
|  |  | 3 | 3 North SEQC – Seq. Controllers (North) | black #13 |  |
|  |  | 4 | 18 IB NE-0 | black #24 (E) |  |
|  |  | 5 | 18 IB NE-1 | black #25 (F) |  |
| South | d0olctl58 | 0 | SEQ/SEQC Power Supplies South | blue AFE 8&9A | 42 |
| North | d0olctl55 | 4 | SEQ/SEQC Power Supplies North | blue #10 | 10 |

****  This entries are NOT quadrant specific , i.e. they  span two quadrants

## Read Out Crates IOC mapping (MCH2 – terminal server  t-d0-mch2)

| VRB crate name | IOC name | IOC port |
| --- | --- | --- |
| 60 | d0olsmt11 | 8 |
| 61 | d0olsmt08 | 30 |
| 62 | d0olsmt13 | 21 |
| 63 | d0olsmt07 | 7 |
| 64 | d0olsmt00 | 28 |
| 65 | d0olsmt06 | 27 |
| 66 | d0olsmt09 | 26 |
| 67 | d0olsmt05 | 6 |
| 68 | d0olsmt02 | 10 |
| 69 | d0olsmt03 | 25 |
| 6A | d0olsmt14 | 31 |
| 6B | d0olsmt04 | 24 |

## HV Crates IOC mapping (MCH2, terminal server t-d0-mch2)

| HV crate name | IOC name  | IOC port |
| --- | --- | --- |
| smts1 | d0olctl08 | 9 |
| smts2 | d0olctl48 | 22 |
| smts3 | d0olctl49 | 23 |
| smts4 | d0olctl25 | 13 |
| smts9 | d0olctl36 | 41 |
| smtn5 | d0olctl23 | 3 |
| smtn6 | d0olctl52 | 4 |
| smtn7 | d0olctl51 | 1 |
| smtn8 | d0olctl53 | 11 |
| smtnA | d0olctl35 | 42 |

## Download GUI

- Contact Harald Fox for any questions about the Download GUI:
- 847-414-4044
- x5222

## SDAQ / Calibrations 

_**Page**_**  **_**Section**_
- Instructions
- Troubleshooting
- Expert Guide

## SDAQ / Calibrations Quick Guide

- Take SMT out of the Global run.
- Start a SEQ only run.
- Make sure the HV is off.
- Set the SVX parameters for read all mode, check the SEQ and SDAQ parameters and download the SMT.
- Power cycle the readout crates and reinit them.
- Bring the HV up.
- Take an SDAQ run and follow the progress on the SDAQ  GUI and the Calibration Manager GUI.
- When data taking is finished switch pulses on. Do not free the trigger!
- Bring the HV down and turn it off.
- Set the SVX parameters to read neighbor.
- Download the detector in sparse readout mode.
- Wait until the data of all crates is committed to the database.
-  Free the trigger, turn pulses off and hand SMT over.
- Make an entry in the log book and in the file ~d0smt/Calibration/CalibrationRuns.txt
- Make a root tuple out of the calibration data.
**For experts only:**
- Process the data to make new thresholds.
- Download the new thresholds.
- Install the new thresholds for examine.
- Transfer the pedestal information to the offline database.

## SDAQ / Calibration

**SDAQ / Calibration**

**Take SMT out of the Global run. **The Calibration run is an SDAQ run. It does not involve or need the L3. It can go on in parallel while experts work on the L3 and it is out of the system. It needs however the L1 system and the COOR / Calibration manager / SDAQ supervisor infrastructure.
**Start a SEQ only run. **The preparation for the SDAQ run will take a while. The best way to ensure that HDI currents stay low is to have a SEQ only run going. While this is strictly speaking not an SDAQ run also this run does not need the L3. See the Section on PDAQ / Taker / Crater for instruction on how to start a SEQ only run.
**Make sure the HV is off. **We need to download the detector. This is only possible when the HV is off.
**Set the SVX parameters for read all mode, check the SEQ and SDAQ parameters and download the SMT:**

- SVX Parameter

- SEQ Parameter

- SDAQ Parameter

- Download crates 60-6B

## SDAQ / Calibration

- This is the button configuration for SDAQ runs. Set this configuration before you download the detector.

- These are the standard configurations. Normally no modification is  needed. Just check.

## SDAQ / Calibration

**Power cycle the readout crates and reinit them. **The VME crates are located in MCH2 and are labeled 0x60 (SMT0_0) through 0x6B (SMT5_1). The power switches are located above the crates. Switch the power off, wait a few seconds and switch it on again. Back in the control room you may have to wait a minute before all crates are back on. Then reinit the crates from the download GUI:
**Bring the HV up. **Check the HV section on how to turn the HV on and on how to ramp it up.

- Reinit crates 60-6B

## SDAQ / Calibration

**SDAQ / Calibration**

**Take an SDAQ run and follow the progress on the SDAQ and calibration manager GUI.**
- Start all necessary GUIs from the All-GUIs-starter:
- Select all the crates in the SDAQ GUI:

- We will need those 4 GUIs for taking and monitoring a calibration run.

## SDAQ / Calibration

- Create a run configuration using the SMT crater GUI:
- End the SEQ all run and choose the new configuration in the Taker window (see the Section PDAQ/Taker/Crater for details).

- 1. Select an SDAQ run

- 2. Make sure that 0x79 and SEQ F are off and all the rest is on.

- 3. Save this configuration

- 4. The file is saved in this location.

## SDAQ / Calibration

- After the SDAQ run is configured the Calibration Manager GUI should look like this:
- Start the Calibration run on the Taker. Follow the progress on the SDAQ GUI. The line “Run Modus” should change from “off” to “init”:
- The “Run Modus” should then change to “ped”, “Total number of events” should read 800 and ”Event number” should steadily increase to 800. Once this is achieved the “Run Modus” should go from “db commit” to “success”. The final result should look like this:

## SDAQ / Calibration

- The Calibration Manager now receives data from the crate. The GUI looks like this:
- At this point the data taking part is finished. The run has been automatically stopped by the Calibration Manager. However, it is important that the trigger is not “freed” in the Taker as long as the commit is not yet finished.
**Start the SMT pulsar from the download GUI.**

## SDAQ / Calibration

**Bring the HV down and turn it off.**
**Set the SVX parameters back to data taking mode: **
**Download the SMT with the new configuration.**
**Wait until all data is committed to the database:**
**Free the trigger in the Taker window and hand SMT back to the DAQ shifter for the global run. **Congratulations! We now have a new calibration run bagged.

## SDAQ / Calibration

**Make an entry in the logbook and in file ~d0smt/Calibration/CalibrationRuns.txt**
**Make a root tuple out of the calibration data. **Go to the directory:
-  cd /home/d0smt/smt_ped_tools_new/runs
- ../smt_ped_tools/scripts/rootFromRun #runnumber
- As summary you will see the number of strips extracted from the database when the process finishes. Sometimes it happens that the process doesn’t exit gracefully. It then consumes loads of memory and CPU time. In this case try to kill the process with an interrupt (^C) or an explicit kill command.

## SDAQ / Calibration Troubleshooting

- The typical problem is the status “error” in the status line of the SDAQ GUI. Stop the run from the Calibration Manager GUI. **Do not stop the run from the Taker.**  Free the trigger (Taker), reboot or power cycle the corresponding crate and try again.
- We had 2 occurrences where above error message would show up permanently for a random crates. The problem was fixed power cycling the 1553 controller crates in MCH3. **Caution: After power cycling those crates a download from scratch is necessary.** The 1553 controllers lose the memory of the power bits. Those need to be written again with a download. For that the HV must be off!
- Sometimes crates are Front End Busy at the beginning of the “ped” stage. In that case a (few) sclinit may help (ask the DAQ shifter). It is advantageous when no other run is started/stopped during the calibration run.

## SDAQ / Calibration Expert Guide

**For Experts only:**
**Process the data to make new thresholds.**
- Start the SDAQ browser from the All-GUIs-starter.

## SDAQ / CalibrationExpert Guide

- Load the SDAQ run you want to investigate. This is usually the most recent run. Watch the text window for error messages. Try again in case of problems.
- Load the reference run. This is usually the last SDAQ run that was used for producing new thresholds. The SDAQ Browser has the ability to compare pedestal and noise for 2 different SDAQ runs.

## SDAQ / CalibrationExpert Guide

- List the problematic HDIs. The program calculates new thresholds from the pedestal and noise data. The SDAQ browser automatically identifies HDIs with different problems and groups them accordingly. One can either look at individual HDIs and decide what to do or implement the new thresholds for a whole group.

## SDAQ / Calibration Expert Guide

- Go through the list of HDIs with big threshold shift and add new thresholds to the database. Select “List HDIs With Big Threshold Shift”, start viewing by pressing the middle control button, and manipulate thresholds by selecting “Input Thresholds”.

- Pedestal and Noise for the reference run

- Pedestal and Noise for the new run

- Pedestal and Noise for the reference run

- Manual threshold input

## SDAQ / CalibrationExpert Guide

- Go through the list of HDIs with many strips below threshold and add new thresholds to the database.
- Commit new thresholds with big threshold difference to the database. There are 2 algorithms implemented to calculate the new threshold: average Pedestal + 6 ADC counts and average pedestal + 3 x average noise. We see a large difference between the methods for noisy HDIs. The automatic calculation for noisy HDIs is fairly good. We can therefore do a global commit for this group.
- Commit new thresholds for noisy HDIs to the database.
- Commit new thresholds for non-problematic HDIs to the database.
**Download the new thresholds. **It is probably best (but not necessary) to restart the download GUI for this.

- Reference run

- New run

- SVX chips

- Manual input

- Threshold in DB

- Commit this threschold and go to next HDI

## SDAQ / CalibrationExpert Guide

**Install the new thresholds for examine. **Select “Global Parameter” and “Offline Cal File” from a download GUI that you don’t need for other purposes. This process takes a lot of time. For each of the crates a output file is written to /online/data/smt/outcal_0x*.txt These files need to be linked to /online/data/smt/cal_0x*.txt Edit the file SmtConfiguration.rcp in /online/examines/pxx.xx.xx/smt_examine/smt_config/rcp/ accordingly.
**Transfer the pedestal information to the offline database. **Follow the instructions in _[http://www-d0.](http://www-d0.fnal.gov/)__[fnal](http://www-d0.fnal.gov/)__[.](http://www-d0.fnal.gov/)__[gov](http://www-d0.fnal.gov/)__[/](http://www-d0.fnal.gov/)_ ~yasuda/smt_transfer.html

## PDAQ / Crater / Taker

- There are only few occasions where a SMT shifter needs to take a PDAQ run. Those are mostly special runs for investigating an SMT specific problem. With some justification you may also count a SEQ only run as PDAQ run. This kind of run does not need the L3 infrastructure. Data is not shipped to the front-end crates. Those runs are useful when SMT cannot be part of the global run or when there is a L3 problem and the SMT needs to be read out to keep the HDI currents stable.
- The tools used to start and control a run are SMT Crater, Taker, Coormon and the DAQ Monitor. They are all started from the All-GUIs-starter:

- Coormon is started from page D: Miscellanous.

## SMT Crater

**SMT Crater. **This GUI is used to configure any of the runs that SMT needs.

- Trigger Selection

- Run Parameters

- Run Mode

- Crate Selection

- Crate Selection Shortcuts

- Save Button

- Status Line

- Run Attributes

## SMT Crater

_**Run Mode:**_ The most important selection.
**PDAQ: **For special runs where the data should go through L3 to tape. Prescale and number of L3 nodes should be adjusted to the needs of the run.
**SDAQ:** This is the run type used for calibration.
**SEQ only:** This run is mainly used to keep the HDI current when there is no global run or SMT does not participate in the global run. This run is especially useful during downloads, when preparing a calibration run. None of the VME readout crates should participate in the run. L3 is not used. This run is close to an SDAQ run.
_**Trigger Selection:**_ One can chose between 3 different L1 triggers:
**Zero bias:** This is the default. The trigger is quasi randomly distributed  (depends on the prescale factor) over any possible bunch crossing.
**Min bias:** Any bunch crossing with interaction is triggered on.
**Tick 10 – BX2:** This is the ticksel2 and triggers only on one specific tick, at the moment tick 10 corresponding to bunch crossing 2. Dan Edmunds can change the tick number easily.
_**Run Parameters: **_
- The **prescale** should be adjusted according to the desired output rate. The zero bias L1 has a rate of 1.7MHz. The default prescale of 50000 gives an output rate of just above 30Hz. The prescale must not be a multiple of 3 or 59, otherwise the selected tick numbers are not quasi random. The prescale can be adjusted later from the Taker.
- The number of **L3 nodes** should be increased depending on how many crates are read out and whether SMT is zero suppressed or not. Talk to the DAQ shifter. If the load on the L3 is too high the readout can get stuck. The number of L3 nodes can be adjusted later from the Taker. The number of L3 nodes has no effect for SEQ only and SDAQ runs.

## SMT Crater

_**Crate Selection:**_ Select here which crates should participate in the special run. Note that 1 SEQ crates corresponds to 2 VRB crates. When a VRB crate is selected the corresponding SEQ crate is selected automatically. If a SEQ only run is chosen the selection is done automatically. Note that crate 0x79 and the corresponding crate SEQ F are located in the test stand in MCH3. Don’t use these crates under normal conditions.
_**Crate Selection Shortcuts:**_ Some buttons to make the crate selection quicker.
_**Run Attributes:**_
**Monitoring:** Adds a monitoring flag to PDAQ runs. The online monitoring is run in parallel on the readout crates.
**Exclusive: **The selected crates are requested exclusively for this run. No other run using these crates can be configured. If a run is already configured with one of the selected crates then configuring the new run will fail. This flag should be set in most cases.
**Auto disable**: This flag may be used to issue single triggers. The run is automatically disabled after a trigger fired. The run can be enabled for a single trigger from the Taker. Currently this function has no use.
_**Save Button:**_ Writes the configuration file.
_**Status Line: **_ Gives the date, time and location of the last configuration file written out. The location is used again for the Taker.

## Taker

- Runs are started and controlled from the Taker GUI:
- Load the configuration file created with SMT Crater. Select <Modify> <Change Trigger...>

- Select the path and filename that was stated in the SMT Crater Status Line. The current values are below. Select <OK>. Check the Taker window for error messages.

## Taker

- Make sure that recording is switched on/off according to your needs (Menu <Modify>). Without recording on (default) the data will not be written to tape.
- You may change the prescales or the number of L3 nodes (<Modify>  <Run Parameters>).
- Start the run with the <Start> button at the bottom of the Taker window.
- Make sure that the number of events written into the status line at the bottom of the Taker window is increasing.
- Stop the run when the desired number of events is reached.
- Free the trigger from the <Modify> Menu.

## High Voltage

_**Page**_**  **_**Section**_
- Description
- Tasks
- Ramping HV up/down
- Global HV Control
- Channel HV Control
- Underdepletion
- HV Alarms
- Troubleshooting
- HV Pod Trips
- HV Crate Trips
- Minor HV Alarm
- Major HV Alarm
- StripTool
- HV Information

## Description of the HV System

- The high voltage (HV) system is used to deplete/bias the sensors
- used in the SMT. The power supplies used for this can be found in
- MCH2. There are a total of 440 HV pods of the BiRa 4877 type in
- three different racks and 10 crates.
- A HV crate consists of a power PC (PPC) for monitoring and control
- and up to 6 HV modules, each with 8 HV pods. The HV is
- distributed via a system of fanout and breakout boxes and the
- interface boards
- SMT sensors are grouped together in HV groups, which can consist
- of up to four sensors. In the case of double-sided sensors, a positive
- and a negative HV pod are serving the HV group.
- The HV pods can be monitored and controlled by a number of GUI's
- in the control room. It is normally not necessary to take any action in
- MCH2.

## Shifter Tasks

- The SMT shifter is responsible for the stable operation
- of the SMT HV system. This encompasses:
- Ramp-up and ramp-down of high-voltage at appropriate times.
- Monitoring of high voltage parameters.
- Monitoring of and reactions on HV alarm conditions.
- Reset of HV trips.
- Documentation of the current HV configuration.
- Documentation of any HV related problems.
- Most of these tasks can be done by using a set of GUI's
- and scripts that are described later.

## When to Ramp HV Up

- HV can be turned **ON **if the detectors are powered under two
- conditions:
- _No Beam_:  If there is no beam of any kind in the Tevatron, then it is safe to bias powered detectors.  Verify with the shift captain that there will be no beam for at least 10 minutes, and that MCR will call before injecting beam.  Biasing is typically done at such times for commissioning or calibration studies like SDAQ runs.
- _Clean, Stable Beam_: This is the case *after* shot setup, injection and scraping have been *successfully* completed.  Check with the shift captain to confirm the status of the beam. Normally the shift captain will give the command to start ramping up the HV.

**When to Ramp HV Down**

- The time needed to ramp down the high voltage should be a little
- more than 3 minutes.
- HV must be turned **OFF** *before*:
- _Shot Injection_:  The detectors cannot be biased when beam is first put into the Tevatron.  During shot setup, check with the shift captain to ascertain the accelerator schedule, and start turning off the HV at least 4 minutes before injection.
- _End of Store_: HV must be turned off before the beam is dumped.  Again, check the schedule with the shift captain, and start turning the HV off several minutes before the end of store. Normally the shift captain will give the command to ramp down the high voltage.
- _Turning on the detector:_ Whenever a HDI  has to be switched on (e.g. after a LV trip), the high voltage corresponding to this HDI has to be ramped down and switched off.
- _Beam Studies_: No HV during any accelerator studies.
- _Beam manipulation during stores_ that might result in higher losses (e.g. rescraping).
- _Work on HV system_: obvious...

## Global HV Monitoring & Control

- To run the global HV GUI:
- Open a terminal window on the upper SMT monitor, and then
- > setup d0online
- > cd /online/config/smt/hv
- > ./smt.hvg &
- Or from the "startgui" gui B.3.

_**Remark:**_ There is no crate B in M222, although there is a power supply for it that is not used.  It might be possible that the DAQ shifter complains about this power supply, since it is in his checklist. It does not affect any SMT operations if this power supply is switched off.

- This is a representation of all the HV crates in MCH2.
- Each HV pod is represented by one of eight pairs of colored boxes under a module number (e.g. “2”) in a crate (e.g. smts9).  A pod’s name is of the form [crate][module][pod][polarity] (e.g. the circled pod is 926P, the polarity (P/N) can be found in the Channel HV gui described later).
- smts1 through smts4 are located in rack M220, smtn5 through smtn8 in M221 and smts9 & smtnA in M222.

## Global HV Monitoring & Control

**Lock**: lock pods at current settings

**Reset**: reset tripped pods

**On**: turn on pods

**Off**: turn off pods

**Reconnect**: reconnect to the monitoring process

**Unlock**: unlock locked pods

**Full**: set and ramp all pods to 100%

**Standby**: set and ramp
- all pods to 0 V

- The right colored box for each pod indicates the pod status:
- Blue: turned OFF
- Orange: turned ON
- Yellow: ramping
- Light Green: OK, ramped up
-    and holding at set voltage
- Dark Green: locked
- Flashing Red: tripped
- Flashing Black: disabled
- The left colored box indicates the pod’s alarm status:
- Light Green: OK
- Yellow: Minor Alarm
- Red: Major Alarm

## Old Global HV                           Monitoring & Control

- To run the old global HV GUI:
- Open a terminal window on the upper SMT monitor, and then
- > setup d0online
- > cd /online/config/smt/hv/OLD
- > ./hv_det_mod.py smtAll.hvd &
- Or from the "startgui" gui B.1.

- This is a representation of all the HV crates in MCH2.
- Each colored rectangle represents one HV pod, and each crate is designated by its name (e.g. smts1).
- smts1 through smts4 are located in rack M220, smtn5 through smtn8 in M221 and smts9 & smtnA in M222.

_**Note:**_ Global HV operations are preferably carried out using the new Global HV GUI (previous section).  The reason that the old Global HV GUI must be running is to provide SMT HV status signals to the Control Room LED status board.

## Old Global HV                    Monitoring & Control

**OFF**: turn off pods

**ON**: turn on pods

**RAMP**: start ramping pods to set voltages

**Set HV**: pull-down menu to define set voltages

**UNLOCK**: unlock locked pods

**LOCK**: lock pods at current settings

**RESUME**: resume paused ramping

**PAUSE**: pause ramping

**RESET**: reset tripped pods

- Pod states are indicated by color:
- Blue: turned OFF
- Orange: turned ON
- Yellow: ramping
- White: paused
- Light Green: OK, ramped up       and holding at set voltage
- Dark Green: locked
- Flashing Red: tripped
- Flashing Black: disabled

## Global HV Ramping Procedure

- Start the Global HV crate GUI
- 	> setup d0online
- 	> cd /online/config/smt/hv
- 	> ./smt.hvg &
- 	Or from the "startgui" gui B.3.
- _Ramp-Up_:
- Turn on all pods: click the **On **button
- Check whether all pods go into the ON-state, or if any are tripped. If any pods are tripped, reset the trips by clicking the **Reset** button.
- Start ramping to full voltage: click the **Full** button
- If any pods trip (i.e. start flashing red), click the **Reset** button immediately, and then click **Full** again.
- When all pods are fully ramped and holding (i.e. light green), lock the pods: click the **Lock** button.
- _Ramp-Down_:
- Unlock all pods: click the **Unlock **button
- Start ramping to 0 V: click the **Standby** button
- When all pods have ramped down to less than 5 V, turn off the pods: click the **Off** button.

## Channel HV Monitoring & Control

- The Channel GUI is particularly useful for working with
- individual pods and troubleshooting (resetting trips,
- tracking currents, etc.).
- A page for each HV crate
- To run this GUI:	> setup d0online
- 			> cd /online/config/smt/hv
- 			> ./smt.hvc
- Or from the "startgui" gui B.2.

## Channel HV Monitoring & Control

**State**: shows current state of the HV pod

**I_Read**: bias current reading

**V_Read**: bias voltage reading

**V_Set**: bias voltage set point

**V_Max**: bias voltage maximum
-             (i.e. bias voltage for full depletion)

**I_Max**: bias current trip point

**V_Trip**: bias voltage hardware trip point

**Reset**: reset tripped pods

**Unlock**: unlock locked pods

**Lock**: lock  pods at current setting

**Resume**: resume paused ramping

**Pause**: pause ramping

**Ramp**: start ramping

**On**: turn on pods

**Off**: turn off pods

**Channel**: HV pod name, last character gives polarity (P/N)

## Channel HV Monitoring & ControlStates & Actions

**Off**: turned off, options from this state are:
- 	- **On**: turn on the pod
- 	- **Disable**: disable the pod
**On**: turned on but not ramped, options from this state are:
- 	- **Off**: turn off the pod
- 	- **Ramp**: ramp the pod to **V_Set**
**Ramp**: ramping to **V_Set**, options from this state are:
- 	- **Off**: turn off the pod
- 	- **Pause**: pause the ramping
**Paused**: ramping has paused, options from this state are:
- 	- **Off**: turn off the pod
- 	- **Resume**: resume ramping to **V_Set**
**Holding**: ramping to **V_Set **completed, options from this state are:
- 	- **Off**: turn off the pod
- 	- **Ramp**: ramp to **V_Set**
- 	- **Lock**: lock the pod at its current setting
**Locked**: locked at current setting, options from this state are:
- 	- **Unlock**: unlock the pod
**CX OC/CX OV/CX**: tripped (overcurrent/overvoltage/group-trip), options from this state are:
- 	 - **Reset**: reset the pod to **On**
**Disabled**: disabled, options from this state are:
- 	- **Enable**: enable the pod
- Pod states can be changed globally using the buttons at the bottom of the GUI,
- or individually by clicking on a pod’s **State** box.

## Underdepletion Script

- Some of the SMT sensors can't hold the full bias voltage they
- were originally designed for. These devices can be biased
- only to a smaller fraction of the original V_max. These
- fractions have already been taken into account when setting
- the current V_max-values. To keep track of underdepleted
- devices, there is a script that prints out a list of these.
- This script is automatically run every time a Begin Run
- checklist or a Configuration Change checklist are entered
- into the logbook.
- The script can be run manually by typing:
- > setup d0online
- > cd ~/monitoring
- > ./list_underdepleted_hdi.py
- A typical output of the underdepletion script is shown on the
- next page.
- Warning: The underdepletion script does not check whether a
- F6-HDI is enabled or not. In case of a switched off F6-HDI,
- the corresponding F8-HDI is without bias as well, since only
- the F6'ers receive the bias for a F-wedge.

## Underdepletion List

- List of underdepleted HV pods as of Sun Apr 21 16:08:43 2002
-         (listed are only sensors that are underdepleted by 5V or more)
-  HDI            planned         current         depleted
-                                     deltaV(V)       deltaV(V)          to      Comment
- ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-         H3-1-21         60              0               0% / 0%     known
-         H3-2-21         60              0               0% / 0%     known
-         H4-1-22         80              71              89% / 110%  known
-         H4-2-22         80              71              89% / 110%  known
-         F12-2-8         80              63              79% / 98%   known
-         H3-2-11         60              23              39% / 52%   known
-         F6-2-5          50              42              85% / 122%  known
-         F6-2-7          50              42              85% / 122%  known
-         H1-1-6          80              72              90% / 111%  known
-         H1-2-6          80              72              90% / 111%  known
-         H2-1-22         75              66              88% / 110%  known
-         H2-2-22         75              66              88% / 110%  known
-         H1-1-18         60              52              87% / 116%  known
-         H1-2-18         60              52              87% / 116%  known
-         F4-2-4          80              71              89% / 110%  known
-         H2-1-24         60              47              79% / 105%  known
-         H2-2-24         60              47              79% / 105%  known
-         H1-1-23         85              67              78% / 95%   known
-         H1-2-23         85              67              78% / 95%   known
-         H1-1-17         80              68              85% / 105%  known
-         H1-2-17         80              68              85% / 105%  known
-         H1-1-13         60              22              38% / 51%   known
-         B3-4-1          70              46              67% / 85%   known
-         B6-7-10         46              34              75% / 112%  known
-         B5-3-1          66              33              51% / 66%   known
-         B5-8-11         46              33              73% / 109%  known
-         H3-1-19         80              55              69% / 85%   known
-         H3-2-24         80              70              88% / 108%  known
-         H3-1-24         80              23              29% / 36%   known
-         H3-2-12         60              30              50% / 67%   known
-         F9-2-8          80              73              92% / 113%  known
-         H3-2-19         80              56              71% / 87%   known
-         H3-1-12         60              22              37% / 50%   known
-         H3-2-18         60              22              38% / 51%   known
-         F6-2-3          50              39              78% / 112%  known
-         F6-2-6          50              23              46% / 66%   known
-         B3-5-3          50              21              42% / 61%   known
-         H2-2-20         60              17              29% / 39%   known
- number of slightly underdepleted sensors (between 5V and 10V below originally planned deltaV)
- ladders:  0
- F-wedges: 4
- H-wedges: 9
- number of significantly underdepleted sensors (more than 10V below originally planned deltaV)
- ladders:  5
- F-wedges: 3
- H-wedges: 17
- WARNING: This list doesn't include switched off F6-HDI's.. In this case the corresponding F8 is not biased as well...

## HV Alarms & Alarm Display

- The SMT HV system is connected to the Significant Event System, which
- means that alarms are being sent if the voltages or currents are
- exceeding normal bounds.
- These alarms can be monitored using the alarm-display. You can start the
- SMT alarm display by typing:
- > setup d0online
- > start_alarm_display -c /home/d0smt/smt.ses &
- Or from the "startgui" gui D.3.
- Each of the buttons in this GUI contains the number of alarms at the different
- severity levels for the corresponding row. The different alarm-levels are the
- following (alarms other than those in row ‘HV’ are not described here):
_**Major alarms:**_
- There is a problem that has to be fixed. For the high voltage, this could mean
- one of the following things:
- HV off: There are ~400 major HV alarms when the HV is turned off. They are voltage alarms for pods for which V_max is greater than ±5V.
- HV trip: A HV pod has tripped of for some reason. In this case you will see in total 4 alarms per tripped pod in the alarm column because of linked alarms.
- Bias-current is very close to the trip-point (less than 5uA).
- Voltage is larger than V_max+5V
- Voltage is smaller than V_max-5V
- A major alarm from a HV trip will also pause a run. A message will come via
- the speakers at the DAQ-shifters console indicating that the run is being paused.

## HV Alarms & Alarm Display

_**Minor alarms:**_
- This is a warning that something might be wrong and the
- corresponding channel should be monitored closely. For the HV, this
- can mean one of the following things:
- HV off: There are ~900 minor HV alarms when the HV is turned off. 864 are related to the *state* of being **OFF **(two alarms for each of the 432 pods).  The rest are minor voltage alarms for pods for which V_max is within ±5V.
- Slightly too high bias-current (within 10uA of trip-point).
- Voltage is larger than V_max+3V
- Voltage is smaller than V_max-3V
_**Invalid alarms:**_
- A read or a write has failed. (Not yet seen for HV alarms !)
_**Disabled Alarms:**_
- It is possible to disable a alarm. It will then not show up again anywhere else
- than in the ‘Disabled’ column, unless it is re-enabled.  Do not disable an alarm
- without a reason.
_**Good Alarms:**_
- If an alarm condition has been resolved, the alarm will go into the GOOD-
- column where it will disappear after 5 minutes. Seeing an increase in the
- number of good alarms without having done something (like ramping the HV
- up) is an indication that there is a alarm-condition that goes away after a short
- time-period. This might be the case if the bias-current is not yet fully at the
- alarm-limit but oscillating around it.
- In the example alarm display on the previous page, all HV pods were off.

## By clicking on the alarm-button, a detailed listing of the alarms will appear.

				          Mark an alarm by clicking on it.
				          Click on SHOW to get detailed 			          information about the marked alarm.
				          Click on DISABLE to disable the 			          marked alarm.
				          Click on DISABLE ALL to 				          disable all alarms listed here.
				          Click on CLOSE to get rid of the 			          window again.

					   In the details window: 
					   Click on CLOSE to get rid 				   of the window.
					   Click on DISABLE to 					   disable the alarm.
					   Click on GUIDANCE to 				   get a message that explains 				   the alarm and proposed 				   actions (not yet implemented).
					   Click on COMMAND to 				   execute a predefined 					   command (not yet 					   implemented).

- By clicking on the alarm-button, a detailed listing of the alarms will appear.
- 				          Mark an alarm by clicking on it.
- 				          Click on **SHOW** to get detailed 			          information about the marked alarm.
- 				          Click on **DISABLE** to disable the 			          marked alarm.
- 				          Click on **DISABLE ALL** to 				          disable all alarms listed here.
- 				          Click on **CLOSE** to get rid of the 			          window again.
- 					   In the details window:
- 					   Click on **CLOSE** to get rid 				   of the window.
- 					   Click on **DISABLE** to 					   disable the alarm.
- 					   Click on **GUIDANCE **to 				   get a message that explains 				   the alarm and proposed 				   actions (*not yet implemented*).
- 					   Click on **COMMAND **to 				   execute a predefined 					   command (*not yet **	**	**	**	**	**   implemented*).

**HV Alarms & Alarm Display**

## TroubleshootingHV Pod Trips

_**Symptoms:**_
- Blinking red field in global HV GUI
- Pod in tripped state in the channel HV GUI
- Major run-pausing alarm in the significant event system.
_**Types of trips:**_
- Overcurrent trip (CXOC):  current exceeded the  bias current trip point.
- Overvoltage trip (CXOV):  voltage exceeded the bias voltage hardware trip point
_**Fix:**_
- For most trips, all that is required is:
- Start monitoring the currents and voltages of the affected pod(s) with StripTool (left-click on the pod name in the channel HV gui).
- Reset the tripped pods: click the **Reset** button
- Ramp the reset pods: click the **Ramp** button.
- Lock the pods when they go into 'holding': click the **Lock** button.
- Note the trip in the logbook together with information about the circumstances of the trip (e.g. tripped while downloading HDI's, trip-indications (CX ??) ...).
_**Risers:**_
- Some pods exhibit a behavior of slowly rising currents.  Eventually, the
- currents for such pods will rise to alarm and trip levels.  They can be identified
- in StripTool by a current trace that is generally steady and high with a slightly
- increasing slope.  If one of these risers trips, it will usually be necessary to
- increase the alarm and trip limits for the tripped pod (and its partner for
- double-sided devices) before returning to long-term stable operation:
- >cd ~/onl_smtnew/util
-        >./increaseHVlim.py &
- Or from the "startgui" gui B.4.

## TroubleshootingHV Pod Trips

- Enter the pod name and new limits
- Click **Increase IMAX by 10 ********A**
- Do not raise a pod by more than 30 A total without checking with the On-call Expert.
- Click **Read set values** if you just want to read back the current value and IMAX
- Click **Quit**
- Note the limit increase in the logbook.
_**Unstable pods:**_
- Some HV trips are caused by unstable channels in the fanout boxes.  These are
- identified by the presence of non-oscillating instabilities of the current as seen
- in the StripTool plot.  These will most likely trip again soon after recovering
- from the first trip.  If a pod trips after it has been identified as unstable:
- Disable the tripped pod (and its partner for double-sided detectors)
- Disable the HDI’s biased by the disabled pods
- Find the associated HDI’s
- 	>cd ~/ monitoring
- 	>show_hvinfo.py <pod name>
- Disable the HDI’s in the download gui
- Have the DAQ shifter start a new run
_**Ramp-up trips:**_
- If a pod trips during ramp-up before it reaches its full voltage,
- Start monitoring the currents and voltages of the affected pod(s) with StripTool (left-click on the pod name in the channel HV gui).
- Try the following solutions ***in order*** until one or  none is successful:
- Ramp in small steps of 10% each to reach 100%.
- If the trip is on overcurrent (CXOC), increase the current trip limit
- >cd ~/onl_smtnew/util                                       >./increaseHVlim.py &
- Or from the "startgui" gui B.4.
- Enter the pod name and new limits

## TroubleshootingHV Pod Trips

- Click **Increase IMAX by 10 ********A**
- Do not raise a pod by more than 30 A total without checking with the On-call Expert.
- Click **Read set values** if you just want to read back the current value and IMAX
- Click **Quit**
- Note the limit increase in the logbook.
- Contact the On-call Expert
- Note the trip in the logbook together with information about the circumstances of the trip (e.g. tripped while downloading HDI's,  trip-indications (CX ??)...).
_**Turn-on Trips:**_
- Sometimes a lot of HV pods immediately trip without ramping then when
- turning a crate **On**. Resets from the global GUI or crate GUI will not reset
- these trips.  To reset pods tripped in this manner:
- Click on the crate name in the global HV GUI.  A detail window will pop up showing all the HV pods in this crate.
- Click on the **Reset** button in this detail GUI.
- Try multiple times if this should not help immediately.
- Note the occurrence in the logbook.
_**IMPORTANT**_
**If the high voltage configuration has to be changed for any reason (e.g. **
**underdepletion or disabling of a pod), a new run must be started to**
**have a known situation on a run-by-run basis.**

## TroubleshootingHV Crate Trips

_**Symptoms:**_ flashing red crate name in the global HV GUI
_**Fix:**_  Before taking any other action, do the following:
- Check the appropriate crate in the crate HV GUI
- If all HV-pods can be seen (try a reconnect first), restart the global HV GUI, since this is a glitch in the GUI.
- Note the occurrence in the logbook
- If this does not help, the HV crate is most likely really tripped. In al
- likelihood the trip is due to a Rack Monitor trip in the HV rack
- (M220, M221, or M222).
- To reset a crate:
- Identify and correct the cause of the crate trip
- Go to MCH2 and check if the RM has tripped.  If so, press the **Reset Alarms** button to reset the RM.
- Click on the flashing crate name in the global HV GUI
- Click **Reset **in the window that pops up.
- Ramp the HV up again if necessary (it is possible that all HV-pods in this crate are off now).
- Note the occurrence in the logbook.

## TroubleshootingMinor HV Alarm

_**Minor HV alarm**_
_**Symptoms:**_ entry in the minor alarm column of the alarm display
_**Fix:**_  It is normal to have ~900 minor HV alarms if the HV is off.  If the minor alarms do not correspond to this case, do the following:
- Find out the alarm-condition, by clicking on the alarm in the alarm-display
- If the alarm is on the bias-current:
- Check in the crate GUI, if this is a real alarm condition (see section on alarm-limits).
- If yes, start monitoring the bias current with StripTool.
- If the alarm is on the bias-voltage:
- Check in the crate GUI, if this is really the case (see section on alarm-limits).
- If yes, start monitoring the voltage with StripTool.
-  Make a note in the logbook.

## TroubleshootingMajor HV Alarm

_**Major HV alarm**_
_**Symptoms:**_ entry in the major alarm column of the alarm display
_**Fix:**_  Do the following:
- First check for tripped HV pods in the global HV GUI.
- Find out the alarm-condition, by clicking on the alarm in the alarm-display
- If the alarm is on the bias-current:
- Check in the crate GUI, if this is really the case (see section on alarm-limits).
- If this HV-pod was already being monitored, and this monitoring indicates that the bias-current will reach the trip-point, the current alarm and trip limits should be increased. Do this as follows:
- >cd ~/onl_smtnew/util                                       >./increaseHVlim.py &
- Or from the "startgui" gui B.4.
- Enter the pod name and new limits
- Click **Increase IMAX by 10 ********A**
- Do not raise a pod by more than 30 A total without checking with the On-call Expert.
- Click **Read set values** if you just want to read back the current value and IMAX
- Click **Quit**
- Note the limit increase in the logbook.

## TroubleshootingMajor HV Alarm

- If this HV-pod was not being monitored before, start a StripTool on it.  If this indicates that the pod will reach the trip-point soon, take the same action as above.
- If the alarm is on the voltage:
- Check in the crate GUI, if this is really the case (see section on alarm-limits).
- If the correspond HV-pod is OFF, turn it on and try to ramp it to 100%.  Turn on the partner pod as well in case of a HV pair.
- If this does not help, call an expert and/or Ben Abraham (*pod malfunction*).
- Note everything in the logbook and run the 'underdepletion' script.

## TroubleshootingStripTool

- 	The StripTool utility allows tracking of bias voltages and currents.  This
- is very useful for debugging problem HV channels.  A StripTool chart can be
- launched from the command line by:
- > setup d0online
- > StripTool &
- In the Control window, enter a signal name (e.g.  SMT_HVC_101P/VOLT,
- SMT_HVC_101P/CURR) and click **Connect** to start the graph.  Plotting
- parameters can be modified in the Control GUI.
- 	StripTool charts for HV pods can also be launched automatically from the
- Crate HV GUI by left-clicking on a HV pod’s name in the **Channel** field.  The
- Control window can be opened again by right-clicking inside the plot-range
- and choosing "Controls Dialog".  It is also advisable to set the ringbuffer size
- to 64000 samples.

## HV InformationFinding HV Info for a Particular Device 

- There are a couple of  ways to obtain HV information for a particular
- channel.  Either of the following may be more useful depending on
- what you are doing.
_**Download GUI**_:  If you are working with the download GUI, then you
- 	can conveniently find HV information for a complete interface board.
- 				                 Click this, select **HV GUI**...
- 					...and get this...
- ...giving HV information and control for each channel on that interface board.
- 		A HV GUI for the entire VRB crate can be obtained by left-clicking on the VRBCR_# button and selecting **HV GUI**.  _**Note:**_ 8-
- 	chippers don't receive bias, so the HV GUI will empty for these.
_**Command line**_:  HV information for a particular detector or IB channel (e.g.
- 	B2-4-6 or SMT_IB_NE0A04-C) can be obtained by:
- 	   >cd ~d0smt/monitoring
- 	   >show_IB_info.py SMT_IB_NE0A04-C
- 		or
- 	   >show_det_info.py B2-4-6
- 	which will produce
- 	a window, which
- 	shows the HV pods
- 	connected to that
- 	detector/channel, and the
- 	other detectors/channels
- 	connected to those pods.

## HV InformationFinding Device Info for a Particular Pod 

_**Command line**_:  A window showing the devices connected to a particular HV
- 	pod (e.g. 133N) can be obtained by:
- 	   >cd ~d0smt/monitoring
- 	   >show_hv_info.py 133

## Alarm Display

- Several portions of the SMT system are connected to the Significant Event
- System.  This means that for those sub-systems, alarms are sent if various
- operating parameters exceed normal bounds.
- These alarms can be monitored using the alarm-display. You can start the
- SMT alarm display by typing:
- > setup d0online
- > start_alarm_display -c /home/d0smt/smt.ses &
- Or from the "startgui" gui D.3.
- Each of the buttons in this GUI contains the number of alarms at the different
- severity levels for the corresponding row. The different alarm-levels are the
- following:
_**Major alarms:**_
- There is a problem that has to be fixed.  Many (but not all) major alarms will
- pause a run. A message will come via the speakers at the DAQ-shifters console
- indicating that the run is being paused.
_**Minor alarms:**_
- This is a warning that something might be wrong and the corresponding system
- should be monitored closely.  The source of the alarm should be identified and
- understood.
_**Invalid alarms:**_
- A read or a write has failed.

## Alarm Display

_**Disabled Alarms:**_
- It is possible to disable an alarm. It will then not show up again anywhere else
- than in the ‘Disabled’ column, unless it is re-enabled.  Consequently, a
- disabled alarm loses the ability to pause a run.  Do not disable an alarm
- without first contacting the Expert On-Call.
_**Good Alarms:**_
- If an alarm condition has been resolved, the alarm will go into the GOOD-
- column where it will disappear after 5 minutes. Seeing an increase in the
- number of good alarms without having done something (like ramping the HV
- up) is an indication that there is a alarm-condition that goes away after a short
- time-period. For example, this might be the case if a bias-current is not yet
- fully at the alarm-limit but oscillating around it.
- Details about diagnosis and treatment for various alarms can be found in the
- SMT Shift Instruction sections for the corresponding sub-systems:
- _Alarm Group Name_	_Shift Instruction Section_
- HV			High Voltage
- LV			Low Voltage Supplies
- IB			Download gui, IB Temperature gui
- Occupancy		Monitoring
- Dead			Monitoring

## By clicking on the alarm-button, a detailed listing of the alarms will appear.

				          Mark an alarm by clicking on it.
				          Click on SHOW to get detailed 			          information about the marked alarm.
				          Click on DISABLE to disable the 			          marked alarm.
				          Click on DISABLE ALL to 				          disable all alarms listed here.
				          Click on CLOSE to get rid of the 			          window again.

					   In the details window: 
					   Click on CLOSE to get rid 				   of the window.
					   Click on DISABLE to 					   disable the alarm.
					   Click on GUIDANCE to 				   get a message that explains 				   the alarm and proposed 				   actions.
					   Click on COMMAND to 				   execute a predefined 					   command.

- By clicking on the alarm-button, a detailed listing of the alarms will appear.
- 				          Mark an alarm by clicking on it.
- 				          Click on **SHOW** to get detailed 			          information about the marked alarm.
- 				          Click on **DISABLE** to disable the 			          marked alarm.
- 				          Click on **DISABLE ALL** to 				          disable all alarms listed here.
- 				          Click on **CLOSE** to get rid of the 			          window again.
- 					   In the details window:
- 					   Click on **CLOSE** to get rid 				   of the window.
- 					   Click on **DISABLE** to 					   disable the alarm.
- 					   Click on **GUIDANCE **to 				   get a message that explains 				   the alarm and proposed 				   actions.
- 					   Click on **COMMAND **to 				   execute a predefined 					   command.

**Alarm Display**

## StripTool

- 	The StripTool utility allows tracking of SMT voltages, currents,
- temperatures and other miscellaneous parameters that are saved as EPICS
- variables.  This is very useful for debugging various problems.  A StripTool
- chart can be launched from the command line by:
- > setup d0online
- > StripTool &
- On the **Curves **page of the Control window, enter a signal name (e.g.
- SMT_HVC_101P/VOLT, see next page) and click **Connect** to start the graph.
- Plotting parameters can be modified in the **Controls **page of the Control
- window.  It is advisable to set the **Ring Buffer** size to 64000 samples.  If the
- Control window is dismissed, it can be opened again by right-clicking inside
- the plot-range and choosing **Controls Dialog**.

## StripToolEPICS Variable Names

- An incomplete list of EPICS variables of the SMT system available for
- StripTool use is given below.

| EPICS Variable Name | Wild Card Completions | Variable Meaning |
| --- | --- | --- |
| SMT_HVC_****/CURR | **** = HV pod name, e.g. 102P, 347N | HV pod bias current |
| SMT_HVC_****/VOLT | **** = HV pod name, e.g. 102P, 347N | HV pod bias voltage |
| SMT_IB_******-*/I_AVDD | ******-* = IB-channel, e.g. SW1B14-A  | IB channel AVDD current |
| SMT_IB_******-*/I_AVDD2 | ******-* = IB-channel, e.g. SW1B14-A  | IB channel AVDD2 current |
| SMT_IB_******-*/I_DVDD | ******-* = IB-channel, e.g. SW1B14-A  | IB channel DVDD current |
| SMT_IB_******-A/TEMP | ****** = IB, e.g. SW1B14  | IB channel A temperature |
| SMT_IB_******-*/TEMP | ******-* = IB-channel, e.g. SW1B14-B | IB channel B-H temperature |
| SMT_IB_******-*/V_AVDD | ******-* = IB-channel, e.g. SW1B14-A  | IB channel AVDD voltage |
| SMT_IB_******-*/V_AVDD2 | ******-* = IB-channel, e.g. SW1B14-A  | IB channel AVDD2 voltage |
| SMT_IB_******-*/V_DVDD | ******-* = IB-channel, e.g. SW1B14-A  | IB channel DVDD voltage |
| SMT_LVSEQ_*/*-+52V-STAT | */* =  side/supply number, e.g. N/2  | sequencer LV power supply +52V status |
| SMT_LVSEQ_*/*-+52VI | */* =  side/supply number, e.g. N/2  | sequencer LV power supply +52V current |
| SMT_LVSEQ_*/*-+52VV | */* =  side/supply number, e.g. N/2  | sequencer LV power supply +52V voltage |
| SMT_LVSEQ_*/*-+5V-STAT | */* =  side/supply number, e.g. N/2  | sequencer LV power supply +5V status |
| SMT_LVSEQ_*/*-+5VI | */* =  side/supply number, e.g. N/2  | sequencer LV power supply +5V current |
| SMT_LVSEQ_*/*-+5VV | */* =  side/supply number, e.g. N/2  | sequencer LV power supply +5V voltage |
| SMT_LV_***/IAVDD-3 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 3-chip AVDD current |
| SMT_LV_***/IAVDD-6 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 6-chip AVDD current |
| SMT_LV_***/IAVDD-89 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 8/9-chip AVDD current |
| SMT_LV_***/IAVDD2 | *** = IB crate power supply, e.g. NE0 | IB LV power supply AVDD2 current |
| SMT_LV_***/IDVDD2-3 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 3-chip DVDD current |
| SMT_LV_***/IDVDD-689 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 6/8/9-chip DVDD current |
| SMT_LV_***/IV15 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 15V current |
| SMT_LV_***/IVCC1 | *** = IB crate power supply, e.g. NE0 | IB LV power supply VCC1 current |
| SMT_LV_***/IVCC2 | *** = IB crate power supply, e.g. NE0 | IB LV power supply VCC2 current |
| SMT_LV_***/IVCC3 | *** = IB crate power supply, e.g. NE0 | IB LV power supply VCC3 current |

## StripToolEPICS Variable Names

- An incomplete list of EPICS variables of the SMT system available for
- StripTool use is given below.

| EPICS Variable Name | Wild Card Completions | Variable Meaning |
| --- | --- | --- |
| SMT_LV_***/MF | *** = IB crate power supply, e.g. NE0 | IB LV power supply magnetic field |
| SMT_LV_***/SPT | *** = IB crate power supply, e.g. NE0 | IB LV power supply shunt plate temperature |
| SMT_LV_***/STAT | *** = IB crate power supply, e.g. NE0 | IB LV power supply status |
| SMT_LV_***/UAVDD-3 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 3-chip AVDD voltage |
| SMT_LV_***/UAVDD-6 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 6-chip AVDD voltage |
| SMT_LV_***/UAVDD-89 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 8/9-chip AVDD voltage |
| SMT_LV_***/UAVDD2 | *** = IB crate power supply, e.g. NE0 | IB LV power supply AVDD2 voltage |
| SMT_LV_***/UDVDD2-3 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 3-chip DVDD voltage |
| SMT_LV_***/UDVDD-689 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 6/8/9-chip DVDD voltage |
| SMT_LV_***/UV15 | *** = IB crate power supply, e.g. NE0 | IB LV power supply 15V voltage |
| SMT_LV_***/UVCC1 | *** = IB crate power supply, e.g. NE0 | IB LV power supply VCC1 voltage |
| SMT_LV_***/UVCC2 | *** = IB crate power supply, e.g. NE0 | IB LV power supply VCC2 voltage |
| SMT_LV_***/UVCC3 | *** = IB crate power supply, e.g. NE0 | IB LV power supply VCC3 voltage |
| SMT_LV_***/VT | *** = IB crate power supply, e.g. NE0 | IB LV power supply Vicor temperature |

## SMT Event Display

- CONTENTS
- How To Run The Program
- The Display
- Modes Of Operation
- Printing Options
- HOW TO RUN THE PROGRAM
- The online SMT Event Display is run with a different command according to the current run type. To run it during normal data-taking, open a new terminal window and type
- > setup d0online
- > start_daq smt_event_display all
- If there is a zero-bias run then use the option 'zb' instead of 'all', ie...
- > start_daq smt_event_display zb
- When the DAQ shifter is taking a special run then you should give 'special' as the argument but if you are unsure of what to use then you can try 'anything'.  This last option receives any data from the first ten runs being taken by the DAQ shifter, regardless of the run type.
- After launching the program, the following windows should be displayed (see next page):

## SMT Event Display

- There are two ways to process events in the SMT Event Display. Selecting 'Start/Resume' from the EventLoop menu (or pressing return in any of the event display windows) will start a continuous event loop. Doing this a second time will pause the event loop. The default time between events is 5 seconds but this can be changed by selecting 'set pause interval' from the EventLoop menu and entering the desired interval, in seconds.
- Selecting 'pause/step' from the EventLoop menu (or pressing space in any of the event display windows) will request the next available event. This allows the user to look at events one at a time.

## SMT Event Display

- THE DISPLAY
- During operation, the SMT Event Display uses eight graphical windows to display data within the silicon detector (as shown below). Six of these are used to display the six barrels and the associated f-disks. One displays the forward f-disks and h-disks. The remaining window contains a key to the barrel display, showing ladder numbers, and a menu for controlling the program.
- Each barrel is displayed in the r-phi plane with the ladders drawn in groups of black or orange, depending on which VRB crate they are connected to. Crate numbers are written on the outside of the barrel.The corresponding f-disk is also drawn on the outside of the barrel with each wedge coloured green. All ladders are pictured as being axially aligned, even though some stereo ladders are offset from the axial plane by 2 degrees or 90 degrees in the real detector. This representation should still be useful for diagnostic purposes.
- The forward disks display window lays all wedges out in a straight line with 12 wedges per f-disk and 24 wedges per h-disk. Again, this representation is geometrically incorrect but diagnostically convenient.
- All detector signals are shown in red for p-side chips and in blue for n-side chips. The length of the line is proportional to the amplitude of the signal recorded, up to a maximum of 256 ADC counts.

## SMT Event Display

- The main window containing the barrel key can be reconfigured by making selections in the 'SMT' menu. The South and North barrel keys and the six barrel readouts can be added / overlaid and removed from the window which allows the user to see 'tracks' which cross more than one barrel.
- MODES OF OPERATION
- The event display runs in one of a number of calibration modes (default is STATIC, explained below) which determine how the data is processed before being displayed. The user can select the desired calibration mode from the 'SMT -> Configurator Mode' menu.These modes are listed below.
- STATIC	The signal as measured by the detector is displayed. A 	global pedestal is also subtracted but this is currently set 	to zero (example shown on previous page.)
- GENERIC 	SDAQ chip pedestals from the database are subtracted 	before the data are displayed (example below.)

## SMT Event Display

- GENERIC_MIXED	GENERIC mode is applied to those VRB crates 	operating in sparse readout mode while DYNAMIC 	mode is applied to those crates operating in all readout 	mode.
- DYNAMIC	Pedestals are calculated on a chip-by-chip basis and 	then subtracted.
- ONLINE_DB	Pedestals are imported from the database on a strip-	by-strip basis and subtracted.
- ONLINE_FILE	Pedestals are imported from a file on a strip-by-strip 	basis and subtracted.
- MIXED	ONLINE_FILE mode is applied to those VRB crates 	operating in sparse readout mode while DYNAMIC 	mode is applied to those crates operating in all readout 	mode.
- PRINTING OPTIONS
- The event display output can be printed to a postscript file by selecting one of the options in the 'File -> PS print' menu. Individual displays can be saved to file or the option exists to save all displays at once, to either a single file or separate files.

## SMT Examine

_**Page**_**  **_**Section**_
- Running SMT Examine
- How to Use Histoscope
- Monitoring Data Quality
- SMT Examine Histograms

## SMT Examine Running SMT Examine

- To run SMT Examine in the Control Room, you should be on _d0ol22_ and logged in as *d0smt*. *(for login do: d0ssh <node name> or ssh -l <user name>  <node name> )*
- Type commands:
**setup d0online**
**start_daq smt_examine runtype**
*	**runtype = ****all**** for physics data taking* examine*	**runtype = ****zb**** for zero bias run**	**runtype = ****special**** for a special SMT run**	**runtype = ****anything**** in the unknown situation*
- In case of any problems, look in the *error.log* file and refer to _[errors and warnings section](http://www-d0.fnal.gov/%7Ealex_k/errors.html)_ to understand what kind of error caused the problem.
- To run _**occupancy monitor**_ type **start_daq smt_occupancy ***  *_Note:_ occupancy monitor runs only together with smt_examine because it uses smt_examine output.

## SMT Examine How to Use Histoscope

- (While reading this section keep referring to Sections 3 and 4)
- A histoscope window looks like the following(Window1):
- If histoscope is running already, such a window should be present.
- You can see a list of histoscope files on the left part of the window. Each file name has the run number in it. An idea of the content of these files can be had by reading Section 3. Barrel, Fdisk, Hdisk files have similar names like
- smt_barrel_occupancy.hs
- smt_fdisk_occupancy.hs

## SMT Examine How to Use Histoscope

- smt_hdisk_occupancy.hs
- Examine uses first 200 events to identify noisy strips and remove them further from cluster reconstruction. During this calibration cluster histograms are not filled. Usually they become available after 15 minutes of running. A shifter should first look at summary_histograms.hs file. These are the only plots that we are interested in saving besides the plots of the devices that you may find causing anomalous “spikes” in these cluster distributions.
- Double click on its name. After this action the histoscope window would like the following(Window 2):

## SMT Examine How to Use Histoscope

- By clicking on the  “ up one level”, you can go back to Window 1.
- Scroll down the window. It has plots of cluster size and cluster charge distributions for ladders/wedges of barrel, fdisk and hdisk. You can double click on the plot names to see them. A histoscope window will pop up showing the clicked plot.
- You can make a multiple seletion also i.e. seeing more than one plot in a single window.  Press “Ctrl” button and select for example  hdisk cluster distributions the cluster charge distributions. After making selection, release the “Ctrl” button and click on “View Multiple” button in the histoscope window. At this instant you do this the histoscope window  would look like the following(Window3):

## SMT Examine How to Use Histoscope

- The result of above step would be a window(Window 4) like( for Hdisk). The contents of the plot this Window 4 are not important. This is just to show how the window looks like.
- You should also make such multiple selection for fdisk and barrel separately. Simply select all the plot in which you find “cluster charge”  and hit  “view multiple” button.
- You should store plots of in  the directory :
- 	/home/d0smt/SHIFTS/*year*/*month*/*date*/*shift*
- 	For e.g.  /home/d0smt/SHIFTS/2002/mar/01/owl
- HOW TO SAVE THE PLOT WINDOW (Window 4) in a file
- Go to the directory
- /home/d0smt/SHIFTS/*year*/*month*/*date*/*shift*
- At the command prompt type(hdisk.jpg is just a file name)
- import  hdisk.jpg
- Go to the “Window 4” and click on the top with middle button of the mouse. You will see that prompt returns in the the window where you typed “ import hdisk.jpg”. See if the file is there.

## SMT Examine How to Use Histoscope

- HOW TO RESET HISTOSCOPE
- 	Double click on “Reset histograms” in Window 1. A very small window like the following pops up.
- 	Left-mouse-hit on it. All histoscope plots would be reset.
- HOW TO LOOK AT THE NUMBER OF EVENTS PROCESSED
- 	Double click on “Events processed” in Window 1. A small window pops up like
- 	Here 30 are the number of events processed.

## SMT Examine How to Use Histoscope

- If you close this window by mistake you can open an x-window and start histoscope by typing
- setup histo
*histo& *
- After typing histo& , histoscope will open like the following window.  From “File/Process” connect to the process running examine. Wait for a while till this window looks like Window 1.

## SMT Examine Monitoring Data Quality

**Summary cluster distributions**
- Open *summary_histograms.hs* and look at _*Summary cluster charge distributions*_ (see SMT Examine Histograms section). Compare histograms with the reference plots. If you do not see significant deviations in shape from the reference plots go to 2.
- If you observe an unusual shape or additional peaks in Landau distributions open *smt_cluster.hs* file and look at the individual ladder/wedge cluster charge distributions for a barrel/disk that shows the problem. Find the device responsible for it and make a log book entry.  Look at the corresponding histogram for the position of the first strip in cluster. Are there any significant peaks?
- Open *smt_barrel_occupancy.hs  *(or the corresponding disk data file)  and look at the occupancy vs event dependence for the ladder/wedge under consideration. Does the device have consistent high occupancy? Make an entry in the log book.
- Open *smt_barrel_main.hs  *(or the corresponding disk data file)  and look at the number of fired strips distribution and the number of entries in each strip for the ladder/wedge under consideration. Does the device have many fired strips in each event? Does the entry histogram show high peaks in some channels? Make an entry in the log book.
- Open *smt_barrel_peds.hs  *(or the corresponding disk data file)  and look at the pedestal vs channel snapshot histograms for the ladder/wedge under consideration. Compare p- and n-side response. Are there correlations between the charge detected on two sides? If no clusters observed check the list of underdepleted ladders:
- cd ~/monitoring setup d0online list_underdepleted_hv.py
- Make an entry in the log book.
- Open *smt_barrel_chips.hs  *(or the corresponding disk data file)  and look at the pedestal distributions for each chip of the ladder/wedge under consideration. If you are running in GENERIC mode (default for smt_examine in the Control Romm) the distributions have to have a peak at zero from neighbour readout and a tail towards higher pulses from the real clusters. Are the chip pedestals consistent with this shape? Is the problem (if any) observed in one chip? Make an entry in the log book.

## SMT Examine Monitoring Data Quality

**Summary entries histograms**
- Open *summary_histograms.hs* and look at _*Summary entries and average pulse histograms *_(see SMT Examine Histograms section). Compare histograms with the reference plots and with the list of disabled HDIs. Make an entry in the log book.
- To make the list:
- as d0smt,
-           		cd ~/monitoring   		./listDisabledHDIs.py

## SMT Examine SMT Examine Histograms

- During usual operation, SMTExamine produces 6 histogram files for each SMT subsystem (barrel, F-disks, H-disks), one summary file and two files containing cluster level plots. This document briefly describes the histograms that can be found in each file.
_**Summary Plots**_
- File *summary_histograms.hs* contains
- summary cluster charge distributions
- summary cluster size distributions
- histograms for number of entries per ladder/wedge
- histograms for average pulse height per ladder/wedge
_***Summary cluster charge distributions***_
- Cluster charge distributions are presented for each barrel. Histograms are split by detector type and side (3-chip ladders, 6-chip ladders p/n side and 9-chip ladders p/n side). Summary charge distributions are expected to have Landau shape. Any deviations from this shape like extra peaks can be usually traced down using detailed cluster plots from *smt_cluster_xxxxx.hs *file (see below _**Cluster plots **_section)
***Barrel cluster histogram names:*** "Bar nb-side ladtype cluster charge" "Bar nb-side ladtype cluster size" where nb - barrel # (1,...,6)       side - p or n       ladtype - L3,L9 (for nb = 1,6); L6,L9 (for nb = 2,3,4,5)
***F-disk cluster histogram names:***** **"FD nd-side cluster charge" "FD nb-side cluster size" where nd - F-disk # (1,...,12)       side - p or n

## SMT Examine SMT Examine Histograms

***H-disk cluster histogram names:***** **"HD nd-side cluster charge" "HD nb-side cluster size" where nd - H-disk # (1,...,4)       side - p or n
_***Summary entries and average pulse histograms***_
- For each barrel/disk side if at least one channel of a ladder/wedge is read out in an event the entry histogram is filled with 1 for this ladder/wedge. If the entry histogram showes zero or very small number of entries for some ladder/wedge the device is either not read out (disabled) or has some problems with calibration or thresholds. For each barrel/disk side for each event average pulse height per ladder/wedge enters the average pulse histogram. Histograms are arranged in the following way: entries for ladder *lad* from layer *lyr* go into channel number *20*(lyr-1)+lad+1 *of the histogram containing all ladders of a particular barrel side. For example, number of entries for ladders 1 through 12 belonging to layer 5 barrel 4 p-side can be found in "Bar 4 p-side ladders entries" histogram in channels 81 through 92. Average pulse height histograms for barrels are arranged similarly. Each F-disk(H-disk) histogram contains entries for wedges 1 through 12 (1 through 24) in channels 1 though 12 (1 through 24).
***Barrel entries and average pulse histogram names:***** **"Bar nb-side ladders entries" "Bar nb-side ladders average pulse" where nb - barrel # (1,...,6)       side - p or n
***F-disk entries and average pulse histogram names:***** **"FD nd all wegdes entries" "FD nb all wedges average pulses" where nd - F-disk # (1,...,12)

## SMT Examine SMT Examine Histograms

***H-disk entries and average pulse histogram names:***** **"HD nd all wegdes entries" "HD nb all wedges average pulses" where nd - H-disk # (1,...,4)       side - p or n
_**Summary cluster charge distributions and summary entries histograms**__ __**are the main histograms that characterize SMT performance, efficiency and data quality. These histograms have to be looked at by a shifter and compared to the reference plots.**__ _
_**Cluster Plots**_
- File  *smt_cluster.hs *contains cluster charge distribution and the position of the first strip of the reconstructed cluster for each barrel ladder and each F- and H- disk wedge (p- and n-side separately).
***Barrel cluster histogram names:*** "Bar nb-lyr-lad-side cluster charge" "Bar nb-lyr-lad-side first strip of cluster" where nb - barrel # (1,...,6)       lyr - layer # (1,..., 8)       lad - ladder # (1,...,12)       side - p or n ***F-disk clusters histogram names:*** "FD nd-nw-side cluster charge" "FD nd-nw-side first strip of cluster" where nd - disk # (1,...,12)       nw - wedge # (1,..., 12)       side - p or n ***H-disk clusters histogram names:*** "HD nd-nw-side cluster charge" "HD nd-nw-side first strip of cluster" where nd - disk # (1,...,4)       nw - wedge # (1,..., 24)       side - p or n

## SMT Examine SMT Examine Histograms

- Cluster charge distribution shapes have to represent Landau distrubution. For most of the ladders/wedges you'll see the peak at small cluster charge indicating the average noise contribution. Any other deviations from the Landau distribution is a signal of the problem.
- Histograms containing the cluster first strip position are filled to ultimately show the noisy strips. If the ladder/wedge does not have any noisy strips the distribution has to be uniform. The peaks in certain channels indicate that this channel initiates fake (noisy) clusters at a rate higher than the real ones.
_**Tick plots**_
- File *tick_histograms.hs* contains cluster charge distribution split by barrel/disk, ladder/wedge type (see above) and side filled for each bunch crossing (tick) separately. These histograms appeared to be useful to make sure that the number of clusters and cluster charge for all bunch crossings are similar.
***Barrel tick histogram names:***** **"Bar nb-side-XX-YY ladtype clusters charge" where nb - barrel # (1,...,6)       side - p or n       ladtype - L3,L9 (for nb = 1,6); L6,L9 (for nb = 2,3,4,5)       XX - superbunch number (X0,X1,X2)       YY - bunch number (01,...,12) ***F-disk tick histogram names:***** **"FD nd-side-XX-YY clusters charge" where nd - disk # (1,...,12)       side - p or n ***H-disk tick histogram names:***** **"HD nd-side-XX-YY clusters charge" where nd - disk # (1,...,4)       side - p or n

## SMT Examine SMT Examine Histograms

_**Occupancy plots**_
- Files *smt_barrel_occupancy.hs* *smt_fdisk_occupancy.hs* *smt_hdisk_occupancy.hs* contain occupancy vs event distributions for each ladder/wedge. Occupancy is defined as a percentage of the channels which were read out per ladder/wedge. The expected average occupancy is about 5%. Occupancy fluctuations above 30%  indicate high common mode noise or many noisy channels on the ladder/wedge. Persistent high occupancy can be a result of the wrong threshold downloaded into the chip.
***Barrel occupancy histogram names:***** **"Bar nb-lyr-lad-side occupancy vs event" where nb - barrel # (1,...,6)       lyr - layer # (1,..., 8)       lad - ladder # (1,...,12)       side - p or n ***F-disk occupancy histogram names:*** "FD nd-nw-side occupancy vs event" where nd - disk # (1,...,12)       nw - wedge # (1,..., 12)       side - p or n ***H-disk occupancy histogram names:***** **"HD nd-nw-side occupancy vs event" where nd - disk # (1,...,4)       nw - wedge # (1,..., 24)       side - p or n
_**Main plots**_
- Files *smt_barrel_main.hs* *smt_fdisk_main.hs* *smt_hdisk_main.hs*

## SMT Examine SMT Examine Histograms

- for each ladder/wedge contain number of entries vs channel number histograms and the distributions of the number of fired strips. First set of histograms allows to track down the channels that are readout more often than the others (noisy channels) while running in srapse mode of the SVX chip. The second set is another way to represent the occupancy.
***Barrel channel entries histogram names:*** "Bar nb-lyr-lad-side entries" "Bar nb-lyr-lad-side fired channels" where nb - barrel # (1,...,6)       lyr - layer # (1,..., 8)       lad - ladder # (1,...,12)       side - p or n ***F-disk channel entries histogram names:***** **"FD nd-nw-side entries" "FD nd-nw-side fired channels" where nd - disk # (1,...,12)       nw - wedge # (1,..., 12)       side - p or n ***H-disk channel entries histogram names:***** **"HD nd-nw-side entries" "HD nd-nw-side fired channels" where nd - disk # (1,...,4)       nw - wedge # (1,..., 24)       side - p or n
_**Pedestal plots**_
- Files *smt_barrel_peds.hs* *smt_fdisk_peds.hs* *smt_hdisk_peds.hs* contain average pedestal vs channel number histograms for each ladder/wedge. They also contain pedestal vs channel histograms updated for each event, in other words, the detector response snapshot. The last set of histogram is useful to observe charge

## SMT Examine SMT Examine Histograms

- correlations between p- and n-side of the same detector when a real particle goes through it.
***Barrel pedestal histogram names:***** **"Bar nb-lyr-lad-side average pulse" "Bar nb-lyr-lad-side event pulse" where nb - barrel # (1,...,6)       lyr - layer # (1,..., 8)       lad - ladder # (1,...,12)       side - p or n ***F-disk pedestal histogram names:***** **"FD nd-nw-side average pulse" "FD nd-nw-side event pulse" where nd - disk # (1,...,12)       nw - wedge # (1,..., 12)       side - p or n ***H-disk pedestal histogram names:***** **"HD nd-nw-side average pulse" "HD nd-nw-side event pulse" where nd - disk # (1,...,4)       nw - wedge # (1,..., 24)       side - p or n
_**Standard deviation plots**_
- Files *smt_barrel_sigma.hs* *smt_fdisk_sigma.hs* *smt_hdisk_sigma.hs* contain sigma of pedestal vs channel number histograms for each ladder/wedge. These histograms are useful to identify the noisy channels (channels with large sigma) of each ladder/wedge. They make sense if data was taken in the _readall_ mode of SVX chip and was processed in _STATIC_ mode (see SmtRawUnp2Data.rcp) .
***Barrel sigma histogram names:***** **"Bar nb-lyr-lad-side std dev" where nb - barrel # (1,...,6)

## SMT Examine SMT Examine Histograms

- 	   lyr - layer # (1,..., 8)       lad - ladder # (1,...,12)       side - p or n ***F-disk sigma histogram names:***** **"FD nd-nw-side std dev" where nd - disk # (1,...,12)       nw - wedge # (1,..., 12)       side - p or n ***H-disk sigma histogram names:***** **"HD nd-nw-side std dev" where nd - disk # (1,...,4)       nw - wedge # (1,..., 24)       side - p or n
_**Chip pedestal distributions**_
- Files *smt_barrel_chips.hs* *smt_fdisk_chips.hs* *smt_hdisk_chips.hs* contain chip pedestal distributions and average chip pedestal vs event for each chip of ladder/wedge. The first set of histograms is used to study total noise. The second set allows to extract common mode noise if data is taken in r_ eadall_ mode of SVX chip and processed in _STATIC_ mode (see SmtRawUnp2Data.rcp) .
***Barrel chip pedestal histogram names:***** **"Bar nb-lyr-lad-side-chip nc pedestal" "Bar nb-lyr-lad-side-chip nc average pulse vs event" where nb - barrel # (1,...,6)       lyr - layer # (1,..., 8)       lad - ladder # (1,...,12)       side - p or n       nc - chip #
***F-disk chip pedestal histogram names:***** **"FD nd-nw-side-chip nc pedestal" "FD nd-nw-side-chip nc average pulse vs event"

## SMT Examine SMT Examine Histograms

- where nd - disk # (1,...,12)       nw - wedge # (1,..., 12)       side - p or n       nc - chip # ***H-disk chip pedestal histogram names:***** **"HD nd-nw-side-chip nc pedestal" "HD nd-nw-side-chip nc average pulse vs event" where nd - disk # (1,...,4)       nw - wedge # (1,..., 24)       side - p or n       nc - chip #
_**Calibration histograms**_
- Files *smt_barrel_cal.hs* *smt_fdisk_cal.hs* *smt_hdisk_cal.hs* contain histograms used to make the list of the noisy channels when smt_examine starts.

## Occupancy GUI

- Contact Harald Fox for any questions about the Occupancy GUI:
- 847-414-4044
- x5222

## MonitoringShifter Instructions

- Silicon detector monitoring is a multi-host system. It's purpose is to
- collect data describing current status of  hardware generate  and send
- alarm messages to Significant Event System.
- Monitoring system consists of the following applications:
-  Linux box multit-threaded application
-  frontend processor VxWorks server
-  Java servlets
_**What to do when**_
**If you need to stop monitoring:**
- login to 'd0ol28' as 'd0run'
- cd '/home/d0run/onlineMonitoring'
- source 'STOP_MONITORING.tcsh'
- check if there are not processes: 'ps -aux | grep SMTDataStore'
- if you see any 'SMTDataStore' kill them:   kill -9 'process_number'
**If you need to start monitoring:**
-  login to 'd0ol28' as 'd0run',
- cd '/home/d0run/onlineMonitoring'
- setup d0online
- check if there are not processes: 'ps -aux | grep SMTDataStore'

## MonitoringShifter Instructions

- if you see any 'SMTDataStore' kill them:   kill -9 'process_number'
- restart monitoring: source START_MONITORING.tcsh
- check if you can access 'http://www-d0ol.fnal.gov/smtMonitoring/'
**If a new message has been sent to Significant Event System:**
- confirm that examine reports the same problem
- get more information about problematic HDI/CHIP:
- using GUI on the left hand side interesting part of the ditector(crate/VBR/HDI) select intersting part of the detector (crate/VRB/HDI)
- using GUI on the right hand side of the 'http://www-d0ol.fnal.gov/smtMonitoring/' select information you need (singnal/hit number/ occupancy)
- look for problems: no events, signal 'zero', high occupancy
**If you think that monitoring does not send information about **
**faulty system to Significant Event System:**
- on the page http://www-d0ol.fnal.gov/smtMonitoring/ check 'IOC DATA UPDATE TIME' and 'SES UPDATE TIME' . Both times should be close to current time. The first one informs when that last data transfer has been done from IOC whereas the second one displays date when the last message has been sent to Significant Event System.
- if data is not collected from an IOC and you are not an expert: PAUSE the run, reboot PowerPC, start monitoring for the IOC from SMT GUI

## MonitoringShifter Instructions

- If data is not collected from an IOC and you are an EXPERT: login to the that IOC, look for suspended processes (VxWork command 'i'), look into log file /home/d0run/onlineMonitoring/IOC_d0olsmtXX_ConnectionLog.txt
- If messages are not sent to Significant Event System and you are not an expert check if SES runs whith no problems, eg. if other application can send data to SES. If only monitoring  cannot talk to SES restart monitoring.
-  If  messages are not sent to Significant Event System and you are an EXPERT check log files /home/d0run/onlineMonitoring/SESLog.txt and /home/d0run/onlineMonitoring/IOC_d0olsmtXX_ConnectionLog.txt. Information is sent to SES only if there is new data comming from IOC. If IOC connections are broken SES messages are suspended.

## MonitoringWhat experts should know

**Linux box application consists of the following threads:**
- IOCConnection thread that collects data from frontend processor servers (actually, that is a set of threads. One thread collects data from a one IOC)
- shmServer thread that serves data to Java servlet queries
- SESConnection thread that sends alarms to Significant Event System
- When application starts it creates data structure to store SMT
- hardware data. Data structure is created based on the current online
- database content. OCI Oracle functions are used to get information
- about all the SMT crates, VRBs, HDIs and chips from online Oracle
- database. Each HDI and chip have tables to discribe signals, hit count
- and occupancy. The 'OCIConnection' thread waits for incomming
- connection requests from frontend processors.  As soon as these
- connection requests appear they are accepted and every 1 minute
- (currently set) data request is sent to frontend processors by
- OCIConnection thread. Received data is stored in previously created
- data stucture. Connections are socket based.
- The 'shmServer' thread serves data to Java servlet requests.
- Connection is socked based. A new socket connection is opened for a
- new data request. Connection is closed as soon as requested data is
- served.  There are two files written to a disk for each data request. An

## MonitoringWhat experts should know

- html file and php file. An html file contains information about how
- many separated images are in the php file. php file contains real data
- and on a base of that file histograms are created on fly in the WWW
- browser.  After a data request has been succesfully completed names
- of existing html files including a new created one are sent to java
- servlet client. Java servlet displayes them in the browser.
- The two threads 'shmServer' and 'OCIConnection' share the same
- data structures. They are synchronized using set of semaphores.
- There is one semaphore created to synchronize access to each SMT
- crate data.
- The 'SESConnection' thread checks hardware data every 10,000
- events (currently set) and generates alarms if occupancy exceeds
- 25% (currently set) or if an HDI or a chip are dead (signal from all
- the strips is zero).  Messages are generated based on the content of
- the same data structure that is used by 'OCIConnection' thread as
- well.  There is an semaphore used to ensure that the data structure is
- not overwitten while check is being done.

## MonitoringWhat experts should know

_**Java servlets**_ (www-d0ol/smtMonitoring) are served by WWW
- server and are used to display the data histograms (signal, hit count,
- occupancy).
- Using the GUI, shown on the left hand side of the WWW page,  you
- can select interesting parts of the detecor. Information can be
- obtained only for those HDIs that are NOT marked as 'disabled'.
- Below the tree like menu of  existing SMT hardware there is a list
- of  'IOC DATA UPDATE TIME' which tells you when the last data
- was collected  from a particular IOC. Based on that time one can
- easily figure out if data base containes updated data for a current run.
- There is 'SES UPDATETIME' list for all IOCs below  already
- mentioned'IOC DATA UPDATE TIME' list. Dates and times in that
- list inform when the last hardware check was done and when the last
- message was sent to Significant Event System.
- The GUI on right hand side of WWW page allows to obtain
- information about SMT hardware in text and graphical formats.

## MonitoringWhat experts should know

- Meaning of selection criteria for text like output:
-  Level  -- what level hardware should be selected on
-  WhatInformation
-  DEAD_HARDWARE -- hardware is concidered dead if there is no signal from any of SVX channels
-  FAULTY_HARDWARE -- hardware is concidered faulty if there is at least one SVX channel with no signal
-  SIGNAL_DATA -- signal mean values
-  HITCOUNT_DATA -- hit number
-  OCCUPANCY_DATA -- occupancy numbers
-  DataSet
-  FULL_SET -- output based on all the collected data from the beginning of the run
- TMP_SET -- output based on tmp data
-  CrateName
-  VRBName
-  HDIName
-  SVXNumber
- Before you push the 'DataAction', you must make your selections.
- The easiest way to fill in the CrateName, VRBName and HDIName
- it to click on a crate, then VRB and then HDI in the left hand WWW
- window.  If you want something at the  SVX level you use the pull
- down menu under SVXNumber.  After 'DataAction' is pushed

## MonitoringWhat experts should know

- selected data will appear in a scrollable widged that is going to be
- drawn below that button.
- To display data in a form of histograms one needs to select/fill  table
- below 'SMT MONITORING GRAPHICS'.
- Meaning of selection criteria for graphics type output.
- Level  -- what level hardware should be selected on
- Type
- HDI_SVX_TYPE -- data for SVXes and for HDIs (adds all SVX's belonging to the HDI)
- HDI_TYPE -- only data for HDI's
- SVX_TYPE -- only data for SVX's
- VRB_TYPE -- some data are available for VRB's
- WhatInformation
- SIG_HIT_GRAPH -- signal and hit distributions
- SIGNAL_GRAPH -- only signal distributions
- HITCOUNT_GRAPH -- onlt hit distributions
- OCCUPANCY_GRAPH -- occupancy plots
- ERROR_GRAPH
- BITPATTERN_GRAPH
- MISSING_BITS_GRAPG
- EVENT_SIZE_GRAPG
- EVENT_BUFF_GRAPG

## MonitoringWhat experts should know

- BUNCH_SIG_HIT_GRAPH
- BUNCH_SIG_GRAPH
- BUNCH_HIT_GRAPH
-  DataSet
- FULL_SET -- output based on all the collected data from the beginning of the run
- TMP_SET -- output based on tmp data
- CrateName
- VRBName
- HDIName
- After button 'DataAction' is pushed selected data will appear in new
- browser widget.
_**Frontend processor server:**_ is a part of SMT frontend processor
- code.  As soon as 'SMT Monitoring' is turned ON in CREATOR
- window SDAQ Supervisor passes all the run commands from COOR
- to frontend processors. At the begin of each run commands INIT and
- START are sent to SMT frontend processor. When INIT command
- comes initialization procedure is called. When START commands
- comes data collection starts.  As soon as initialization is completed
- monitoring server tries to connect to Linux box application
- (IOCConnection thread) .  At that point IOCConnection process

## MonitoringWhat experts should know

- takes over and asks server for data every 1min (currently
- set).  Connection is socket based.  As soon as run is stopped
- command STOP is passed via  SDAQ Supervisor to frontend and
- monitoring server breaks connection.
- Hardware data for monitoring purposes can be collected from:
- VRB spy memory (A24/D16 transfer). At a time on a request only  data from one HDI is stored in VRB spy memory.  Handshake is described in VRB documentation. That procedure is very slow and it is not currently used.
- onl_smtcalib/ssdaq/smtStore/monitoringbuffer and an interrupt is generated. That interrupt is cought by PowerPc and data is copied via VME backplane.  That procedure is very efficient and is currently used.
- Collected hardware data is unpacked and stored in to histograms for
- each HDI and each chip.  On a request from IOCConnection thread
- that data is copied to a buffer and sent back to IOCConnection
- thread.
_**Current setup**_.
- The whole software is stored in cvs repository in the following
- directories:
- onl_smtcalib/ssdaq/smtStore/monitoring -- Linux box C++ code
- onl_smtcalib/ssdaq/smtStore/servlet  --  Linux box Java servlets
- onl_smtcalib/ssdaq/smtStore/frontend  --  frontend C code for VXWorks

## MonitoringWhat experts should know

- SMT online monitoring application runs on 'd0ol28.fnal.gov' D0
- online host from 'd0run' account.  Application is located in directory
- '~d0run/onlineMonitoring'.
- Since graphs are created using PHP software PHP library needs to be
- supported by WWW server.  Current version installed is 'php-4.1.2'.
- For graphs creation an additional packed 'jpgraph-1.6.1' (written is
- PHP) was used. It is installed on a disc accessible from online cluster
- in directory `/d0usr/products/jpgraph'.  There are links created in the
- application to the above mentioned location.
- SMT online monitoring application should be started/ stopped using
- official d0 scrit start_daq/stop_daq  Usage of that script ensures that
- only one instance of application runs at a time.
- There are several log files created in dir `~d0run/onlineMonitoring/'
- IOC_d0olsmtXX_ConnectionLog.txt  -- stores information about data collection from SMT IOCs
- DataServerLog.txt -- stores information about requests from Java servlets
- SESLog.txt -- stores information about hardware checks and messages sent to Significant Event System
- Every time application is stopped log files are copied to
- '/projects/smtMonitoring/log/' directory with current date and time.
- All 'html' and 'php' files that contain created graphs are in directory
- '/projects/smtMonitoring/jpgraph_cache'.

## MonitoringWhat experts should know

- Files older then 1 month are purged from both directories
- '/projects/smtMonitoring/log/' and
- '/projects/smtMonitoring/jpgraph_cache' by an automatic cron job.
_**Compilation and Linking.**_
- In order to compile C++ code
- ('onl_smtcalib/ssdaq/smtStore/monitoring') one needs to 'setup
- D0RunII p13.10.00' or later if backward compatible.
- In order to compile Java servlets
- ('onl_smtcalib/ssdaq/smtStore/servlet') one needs to:
- 'setup tomcat' and 'setup java v1_3_1_02'.  Compilation is done
- automatically by 'makefile'.  Copiled files 'class' are copied to
- `www/WEB-INF/classes/' and library 'jar' is copied to 'www/WEB-
- INF/lib/'. After successful compilation one needs to copy 'class' and
- 'jar' files to a well known location of 'tomcat' server. Currently that
- are:
- '/projects/elog/jakarta-tomcat-3.2.1/webapps/smtMonitoring/WEB-
- INF/classes/' and
- '/projects/elog/jakarta-tomcat-3.2.1/webapps/smtMonitoring/WEB-
- INF/lib/'
- In order to compile frontend part change directory to
- 'onl_smtcalib/ssdaq/smtStore/frontend' and use existing makefile that
- will make the whole job for you.

## The Archivers

_**Page**_**  **_**Section**_
-  Introduction
-  Once a shift tasks
- Fast archiver check
- Slow archiver check
- Radmon archiver check
-  Once a week tasks (Friday day shift)
- Save slow archiver data
- Save radmon archiver data
- Cleanup and restart fast archiver
-  Plotting archiver data

## The ArchiversIntroduction

- Philosophy
-  What gets saved and why :
- We have three processes of the Archiver running, they record periodically data gathered by the control system (via the 1553 link).
- The fast Archiver records data  at 1 Hz rate and  this data is primarily used to diagnose transients that may cause us problems. This system has proven invaluable in diagnosing power supply glitches that have plagued us in the past. This data is of no long term importance and thus only a week’s worth of stored data is saved on a scratch disk. Every  week we stop the process, clean up the stored data and restart it.
- The slow Archiver records parameters at a much slower rate (from a few times a minute to once an hour) that may be of some importance in monitoring the long term performance of the detector – these are primarily the bias voltages (HV) of the HDI’s, their power consumption,

- The procedures outlined here are to be carried out either every shift (checking and restarting the Archivers, if required), or once a week, during the Friday day and/or evening shifts (saving the Archiver data and restarting the Archivers)
-  Written  June 5, 2003  P. Rapidis

## The ArchiversIntroduction

- 	and environmental variables (temperatures, water flows, magnetic fields ...). These pieces of data get saved for long term perusal; they get transferred onto tapes via the SAM system. The transfer onto the SAM data tapes, the erasing of the old data, and the restarting of the process is also done once a week. If for some reason the slow Archiver gets stopped (e.g. it terminates abnormally) earlier in a the week,  then the recovery procedure should be a transfer of the older data to SAM followed by  a restarting of the process (I.e. follow the procedure for the weekly saving of data for this process).
-      The Radmon Archiver  records parameters related to the radiation loss detectors (the fingers) and to the BLM’s at the end of the D0 detector, both at a low rate (once a minute for temperatures and integrated doses) and at a fast rate (1 Hz, for instatneous loss rates) If for some reason the Radmon Archiver gets stopped (e.g. it terminates abnormally) earlier in a the week,  then the recovery procedure should be a transfer of the older data to SAM followed by  a restarting of the process (I.e. follow the procedure for the weekly saving of data for this process).

## The ArchiversIntroduction

- A detailed list of variables being saved:
-        Fast Archiver:
- 	_[http://d0server1.](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)__[fnal](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)__[.](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)__[gov](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)__[/projects/silicon/www/](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)__[Archiver](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)__[/](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)__[fastlist](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)__[.](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)__[htm](http://d0server1.fnal.gov/projects/silicon/www/Archiver/fastlist.htm)_
- 	Slow Archiver:                       _[http://d0server1.](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)__[fnal](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)__[.](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)__[gov](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)__[/projects/silicon/www/](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)__[Archiver](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)__[/](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)__[slowlist](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)__[.](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)__[htm](http://d0server1.fnal.gov/projects/silicon/www/Archiver/slowlist.htm)_
-          Radmon Archiver
-          (ask the radiation loss experts)
- _A general note_ – at some steps you will be asked to see if a file has a current date. Please be cognizant of the fact that the Archiver (esp. the slow Archiver) may be up to a few minutes late in updating the output files. So be somewhat generous when interpreting the date stamp information

**The Archivers****Introduction**

## The ArchiversTo be done once a shift

**The Archivers****To be done once a shift**

-  _Make sure the fast Archiver is running_
-       It is assumed that you are signed on as d0smt on an online machine and you have done a >setup d0online
-  Either you can issue the command
- >d0ssh d0ol28 ‘ps –ef | grep d0smt | grep archi | grep –v grep’
- 	and check that you see three jobs running;
- 	also check if the archive file is updating by
-  >d0ssh d0ol28 ‘ls –l /scratch/smt/psarchive | grep
- 			`date +%Y%m%d`’
- 	and check if the file with a name like 200ymmdd-000000 (where ymmdd are today’s year, month, and date) has a current time stamp.
-    or execute the script:
- >/home/d0smt/archiverchecks.sh
- 	which does the above (and more) for you
-  If things look OK go on to checking the slow Archiver.

**Easier step**

## The ArchiversTo be done once a shift

-  If the fast Archiver is dead you should kill any malfunctioning jobs, clean up the lock file, and restart it. This is done by doing :
- 	Sign on d0ol28 as d0smt by doing :
- >d0ssh d0ol28    if needed
- >setup d0online	  if needed
- >setup chan_archiver
- >stop chan_archiver –p 4820
- >cd /scratch/smt/psarchive
- >rm archive_active.lck  done if this file exists
- 	kill any leftover jobs found in step 1 via
- >kill –9 <*pid*>  get the process id number from step 1
- 	restart the fast Archiver by doing
- >start_chan_archiver –p 4820 –w **.** –c ps_config
-          You must be in the /psarchive directory,
-           and just for your information 4820 is the port number
- 	 check (after a couple of minutes) if the fast Archiver is running by repeating the tests of step 1.

## The ArchiversTo be done once a shift

**Easier step**

**The Archivers****To be done once a shift**

- _Make sure the slow Archiver is running_
-  Either you can issue the command
- >d0ssh d0ol03  ‘ps –ef | grep d0smt | grep archi | grep –v grep’
- 	and check that you see three jobs running;
- 	also check if the archive file is updating by
-  >d0ssh d0ol03 ‘ls –l /projects/archive/smt/current | grep
-         `date +%Y%m%d`’
- 	and check if the file with a name like 200ymmdd-000000 (where ymmdd are today’s year, month, and date) has a current time stamp.
-    or execute the script:
- >/home/d0smt/archiverchecks.sh
- 	which does the above (and more) for you.
-  If things look OK then you are done, go do something else now !
- If the slow Archiver is dead, then follow the procedure used to back up the slow Archiver data (which should be done once a week, _**or**_ in an exceptional circumstance, such as the one you are facing now).

## The ArchiversTo be done once a shift

- _Make sure the Radmon Archiver is running_
-  Either you can issue the command
- >d0ssh d0ol23  ‘ps –ef | grep d0rad | grep archi | grep –v grep’
- 	and check that you see three jobs running;
- 	also check if the archive file is updating by
-  >d0ssh d0ol23 ‘ls –l /projects/archive/radmon/current | grep
-         `date +%Y%m%d`’
- 	and check if the file with a name like 200ymmdd-000000 (where ymmdd are today’s year, month, and date) has a current time stamp.
-    or execute the script:
- >/home/d0smt/archiverchecks.sh
- 	which does the above (and more) for you.
-  If things look OK then you are done, go do something else now !
- If the Radmon Archiver is dead, then follow the procedure used to back up the Radmon Archiver data (which should be done once a week, _**or**_ in an exceptional circumstance, such as the one you are facing now).

**Easier step**

**The Archivers****To be done once a shift**

## The ArchiversTo be done once a week

-  	These procedures are to be done once a week (Friday day and/or evening shift ).
- 	Steps I and II  will also be needed in the situation of a malfunctioning slow/Radmon Archiver.
- 	The are five distinct operations here.
-  	Ia and Ib) We save the data of the slow/Radmon Archiver by copying the data to a temporary location, *tar*ing it and *gzip*ing it. The original directory is cleared and the slow/Radmon Archiver is restarted.
- 	IIa and IIb) The *tar/gzip*ed file is transferred to SAM, and upon a successful SAM transfer the *tar/gzip*ed file is deleted. (These two operations are to be done once a week and in the case of a malfunctioning slow/Radmon Archiver).
- 	III) In addition, for the fast Archiver, we delete the older data of the fast archiver from a backup directory, copy its current data to the backup directory, clean up the current data directory and restart the fast Archiver.

## The ArchiversTo be done once a week

- _Save the data from  the slow Archiver _
- Login and setup :
- 	Login as d0smt on d0ol03, this can be done only from d0ol00 and d0ol05 by doing
- >setup d0online    if needed
- >d0ssh d0ol03    if needed, from d0ol00/d0ol05 only
- >setup d0online	  if needed
- >setup chan_archiver
- >cd /projects/archive/smt
- Copy and compress the older data
- >chan_arch_copy smt
- 	This copies the data from the previous week (from *./previous*) to a temporary location (*/d0olcluster/scratch*) and then *tar*s and *gzip*s it. This can easily take ~1hour for a week’s worth of data. If you are compulsive you can check the progress of this step by doing the following on another x-terminal :
- >ls –l /d0olcluster/scratch/archive/smt/dir*
- >ls –l /projects/archive/smt/previous
- 	Please wait for this compressing to end before proceeding

## The ArchiversTo be done once a week

- Stop the slow Archiver process
- >stop_chan_archiver –p 4815  if needed,
- 	if you are here because the slow Archiver malfunctioned you may not need this, but
- 	it does no harm to do it.
- 	Check if the three jobs running have stopped. Sometimes two of the three jobs take 5 to 10 minutes to die. If they take too long you may want to kill them. Check on the jobs by doing
- >ps -ef | grep d0smt
- 	and, if needed, kill them by :
- >kill –9 <*pid*>  where you use the process id
- 			from the previous command
- 	If you kill the jobs, or if things have ended abnormally you will have to remove the lock file by doing:
-  >rm ./current/archive_active.lck
- Copy the current data
- 	Copy the current data (from ./current) to the backup location (to ./previous)
- >rm –Rf ./previous/*
- >chan_arch_move smt  for this to work there should be no *./current/archive_active.lck*  file

## The ArchiversTo be done once a week

**The Archivers****To be done once a week**

- Start the slow Archiver once again
- >cd ./current  (I.e. go to /projects/archive/smt/current)
- >start_chan_archiver –c config_file –w **.** –p 4815
- 		make sure the slow Archiver is running, this 	can  take a few minutes to complete. You can 	check this by  either issuing the command
- >ps –ef | grep d0smt
- 	and check that you see three jobs running;
- 	also check if the archive file is updating by
-  >ls –l /projects/archive/smt/current  | grep  ‘date +%Y%m%d’
- 	and check if the file with a name like 200ymmdd-000000 (where ymmdd are today’s year, month, and date) has a current time stamp.
- Or you can do it  by executing  the script:
- >/home/d0smt/archiverchecks.sh
- 	which does the above (and more) for you.
- 	You may log  out of d0ol03 now.

**Easier Step**

## The ArchiversTo be done once a week

- 6. Transfer the compressed data to SAM
- (More details can be found in the file */projects/archive/HOWTO_SAM*)
-      Log on to d0olb as d0smt, this can be done if you are d0smt on one of the following three machines d0ol44,d0ol28, and d0ol05
- >setup d0online  if needed
- >d0ssh d0olb  from d0ol44,d0ol28, and d0ol05 only !
- 	go to the directory where the compressed data is (from step 2) I.e. :
- >cd /scratch/archive/smt/dir<tab>   this is another name for the directory */d0olcluster/scratch/archive/...* of step 2
- 	Create and edit the metafile that controls the transfer to SAM
- >setup nedit
- >../sammfile
- 	An edit window (nedit) will pop up in which you should edit the five lines with the *data_file, parameter_file, start_date, end_date*, and *file_size*. Please note that (a) you should change the filenames on some of these five lines (including the date part, ***_****ctl****_*** to **_***smt***_**, etc.- paste the names from the x-terminal) and (b) the size of the *tar.gz* file as they were shown on your x-terminal.

## The ArchiversTo be done once a week

- 	Exit the editor and save the file :	online_epics_to_sam.py
- 	Setup SAM by doing :
- 	>source /projects/archive/SAM_SETUP.sh
- 	 Check that things are setup properly :
- 	>which sam  should be V3_1_9a or greater
- 	>echo $SAM_STATION   should be datalogger-d0olb
- 	 Submit the job to SAM:
-  	>sam store  - -descrip=online_epics_to_sam.py  - -source=**.**
- 	or if you encounter an error you can resubmit it via:
-  >sam store  - -descrip=online_epics_to_sam.py  - -source=**.**
-                   - -resubmit
- (note that there is no space between the two dashes in these commands)
-       Now wait for the completion of the data transfer (usually less than 15 minutes).
- 	If the transfer is successful remove the old data.
- 	First check the file transfer
- 	>setup netscape
- 	>netscape &   and go to            _[http://d0db.](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[fnal](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[.](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[gov](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[/](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[sam](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[_data_browsing/data_file_query.html](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)_
- 		which is also bookmarked as “SAM data browser”
- 		use datatier = *epics*  ,  datafile = *dir_smt%*
- 		application family = *online_archive  , *and
- 		application_name= *online_epics*

## The ArchiversTo be done once a week

- 	and see if the file from step 2 was stored properly. If this does not work out, try a resubmit as outlined above, and if this even fails notify an expert (P. Rapidis or V. Sirotenko)  and do not do the cleanup of the next step.
- Delete the old files (to be done if all is well from the previous step)
- 	>cd /scratch/archive/smt/dir<tab><carriage return>
- 	>rm *
- 	>cd ..
- 	>rmdir dir<tab><carriage return>
- 	>ls  you should see only one file named “sammfile”
- 	At this point log off d0olb.
-        If you are doing the above as a result of a malfunction of the slow Archiver, then please skip the next steps – you are done with the Archiver tasks. If this is part of the weekly task then continue.

## The ArchiversTo be done once a week

- _Save the data from  the Radmon Archiver _
- Login and setup :
- 	Login as **d0rad **on d0ol23, this can be done by doing
- >setup d0online    if needed
- >d0ssh –l d0rad d0ol23
- >setup d0online
- >setup chan_archiver
- >cd /projects/archive/radmon
- Copy and compress the older data
- >chan_arch_copy radmon
- 	This copies the data from the previous week (from *./previous*) to a temporary location (*/d0olcluster/scratch*) and then *tar*s and *gzip*s it. This can easily take ~1hour for a week’s worth of data. If you are compulsive you can check the progress of this step by doing the following on another x-terminal :
- >ls –l /d0olcluster/scratch/archive/radmon/dir*
- >ls –l /projects/archive/radmon/previous
- 	Please wait for this compressing to end before proceeding

## The ArchiversTo be done once a week

- Stop the radmon Archiver process
- >stop_chan_archiver –p 4813  if needed,
- 	if you are here because the Radmon Archiver malfunctioned you may not need this, but
- 	it does no harm to do it.
- 	Check if the three jobs running have stopped. Sometimes two of the three jobs take 5 to 10 minutes to die. If they take too long you may want to kill them. Check on the jobs by doing
- >ps -ef | grep d0rad
- 	and, if needed, kill them by :
- >kill –9 <*pid*>  where you use the process id
- 			from the previous command
- 	If you kill the jobs, or if things have ended abnormally you will have to remove the lock file by doing:
-  >rm ./current/archive_active.lck
- Copy the current data
- 	Copy the current data (from ./current) to the backup location (to ./previous)
- >rm –Rf ./previous/*
- >chan_arch_move radmon for this to work there should be no *./current/archive_active.lck*  file

## Start the radmon Archiver once again
>cd ./current  (I.e. go to /projects/archive/radmon/current) 
>start_chan_archiver –c config_file –w . –p 4813
		make sure the radmon Archiver is running,    	this can  take a few minutes to complete. You 	can check this by  either issuing the command 
>ps –ef | grep d0rad  
	and check that you see three jobs running;
	also check if the archive file is updating by
 >ls –l /projects/archive/radmon/current  | grep  ‘date +%Y%m%d’
	and check if the file with a name like 200ymmdd-000000 (where ymmdd are today’s year, month, and date) has a current time stamp. 
Or you can do it  by executing  the script:
	>/home/d0smt/archiverchecks.sh 
	which does the above (and more) for you.

	You may log  out of d0ol23 now.

- Start the radmon Archiver once again
- >cd ./current  (I.e. go to /projects/archive/radmon/current)
- >start_chan_archiver –c config_file –w **.** –p 4813
- 		make sure the radmon Archiver is running,    	this can  take a few minutes to complete. You 	can check this by  either issuing the command
- >ps –ef | grep d0rad
- 	and check that you see three jobs running;
- 	also check if the archive file is updating by
-  >ls –l /projects/archive/radmon/current  | grep  ‘date +%Y%m%d’
- 	and check if the file with a name like 200ymmdd-000000 (where ymmdd are today’s year, month, and date) has a current time stamp.
- Or you can do it  by executing  the script:
- 	>/home/d0smt/archiverchecks.sh
- 	which does the above (and more) for you.
- 	You may log  out of d0ol23 now.

**The Archivers****To be done once a week**

**Easier step**

## The ArchiversTo be done once a week

- 6. Transfer the compressed data to SAM
- (More details can be found in the file */projects/archive/HOWTO_SAM*)
-      Log on to d0olb as d0rad, this can be done if you are d0smt on one of the following three machines d0ol44,d0ol28, and d0ol05
- >setup d0online  if needed
- >d0ssh –l d0rad d0olb  from d0ol44,d0ol28, and d0ol05 only !
- 	go to the directory where the compressed data is (from step 2) I.e. :
- >cd /scratch/archive/radmon/dir<tab>   this is another name for the directory */d0olcluster/scratch/archive/...* of step 2
- 	Create and edit the metafile that controls the transfer to SAM
- >setup nedit
- >../sammfile
- 	An edit window (nedit) will pop up in which you should edit the five lines with the *data_file, parameter_file, start_date, end_date*, and *file_size*. Please note that (a) you should change the filenames on some of these five lines (including the date part, ***_****ctl****_*** to **_***radmon***_**, etc.- paste the names from the x-terminal) and (b) the size of the *tar.gz* file as they were shown on your x-terminal.

## The ArchiversTo be done once a week

- 	Exit the editor and save the file :	online_epics_to_sam.py
- 	Setup SAM by doing :
- 	>source /projects/archive/SAM_SETUP.sh
- 	 Check that things are setup properly :
- 	>which sam  should be V3_1_9a or greater
- 	>echo $SAM_STATION   should be datalogger-d0olb
- 	 Submit the job to SAM:
-  	>sam store  - -descrip=online_epics_to_sam.py  - -source=**.**
- 	or if you encounter an error you can resubmit it via:
-  >sam store  - -descrip=online_epics_to_sam.py  - -source=**.**
-                   - -resubmit
- (note that there is no space between the two dashes in these commands)
-       Now wait for the completion of the data transfer (usually less than 15 minutes).
- 	If the transfer is successful remove the old data.
- 	First check the file transfer
- 	>setup netscape
- 	>netscape &   and go to            _[http://d0db.](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[fnal](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[.](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[gov](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[/](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[sam](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)__[_data_browsing/data_file_query.html](http://d0db.fnal.gov/sam_data_browsing/data_file_query.html)_
- 		which is also bookmarked as “SAM data browser”
- 		use datatier = *epics*  ,  datafile = *dir_radmon%*
- 		application family = *online_archive  , *and
- 		application_name= *online_epics*

## The ArchiversTo be done once a week

- 	and see if the file from step 2 was stored properly. If this does not work out, try a resubmit as outlined above, and if this even fails notify an expert (P. Rapidis or V. Sirotenko)  and do not do the cleanup of the next step.
- Delete the old files (to be done if all is well from the previous step)
- 	>cd /scratch/archive/radmon/dir<tab><carriage return>
- 	>rm *
- 	>cd ..
- 	>rmdir dir<tab><carriage return>
- 	>ls  you should see only one file named “sammfile”
- 	At this point log off d0olb.
-        If you are doing the above as a result of a malfunction of the radmon Archiver, then please skip the next steps – you are done with the Archiver tasks. If this is part of the weekly task then continue.

## The ArchiversTo be done once a week

- _Cleanup and restart of the fast Archiver_
- Login and setup :
- 	Login as d0smt on d0ol28, this can be done only from d0ol00, d0ol03, and d0ol05 by doing
- >setup d0online    if needed
- >d0ssh d0ol28  	  if needed, from d0ol00/d0ol05 only
- >setup d0online	  if needed
- >setup chan_archiver
- >cd /scratch/smt/
- Delete the older files
- >rm –Rf ./old/*
- Stop the fast Archiver process
- >stop_chan_archiver –p 4820
- 	Check if the three jobs running have stopped. Sometimes two of the three jobs take 5 to 10 minutes to die. If they take too long you may want to kill them. Check on the jobs by doing
- >ps-ef | grep d0smt | grep arch
- 	and, if needed, kill them by :
- >kill –9 <*pid*>  where you use the process id from the previous command

## 	If you kill the jobs, or if things have ended abnormally you will have to remove the lock file by doing:
 >rm ./psarchive/archive_active.lck
Move the current files (from ./psarchive)  to the older file directory (to ./old) by:
 >mover  moves the files and creates new directories in ./old 
Restart the fast Archiver
>cd ./psarchive
>start_chan_archiver –p 4820 –w . –c ps_config
Check that the fast Archiver is running 
		this can  take a few minutes to complete. 	You can check this by  either issuing the 	command 
>ps –ef | grep d0smt  | grep archi
	and check that you see three jobs running;
	also check if the archive file is updating by
 >ls –l /scratch/smt/psarchive  | grep  ‘date +%Y%m%d’
	and check if the file with a name like 200ymmdd-000000 (where ymmdd are today’s year, month, and date) has a current time stamp. 	                                                        Or you can do it via the script:
>/home/d0smt/archiverchecks.sh 
	which does the above (and more) for you.    THE END

- 	If you kill the jobs, or if things have ended abnormally you will have to remove the lock file by doing:
-  >rm ./psarchive/archive_active.lck
- Move the current files (from *./psarchive*)  to the older file directory (to *./old*) by:
-  >mover  moves the files and creates new directories in ./old
- Restart the fast Archiver
- >cd ./psarchive
- >start_chan_archiver –p 4820 –w **.** –c ps_config
- Check that the fast Archiver is running
- 		this can  take a few minutes to complete. 	You can check this by  either issuing the 	command
- >ps –ef | grep d0smt  | grep archi
- 	and check that you see three jobs running;
- 	also check if the archive file is updating by
-  >ls –l /scratch/smt/psarchive  | grep  ‘date +%Y%m%d’
- 	and check if the file with a name like 200ymmdd-000000 (where ymmdd are today’s year, month, and date) has a current time stamp. 	                                                        Or you can do it via the script:
- >/home/d0smt/archiverchecks.sh
- 	which does the above (and more) for you.    **THE END**

**The Archivers****To be done once a week**

**Easier Step**

## The ArchiversPlotting archiver data

_** **_
- For devices being archived by the **slow** archiver use any of the
- online machines and do :
**> **setup d0online
- > setup chan_archiver
- > Xarr.py  /projects/archive/smt/xxxxxx/dir.200ymmdd-hhmmss &
-       ( xxxxxx can be either  ‘current’ or ‘previous’, and y,mm,dd,hh,mm,ss stand for the year, month, day, hour, minute, and second  when this archive file started)
- For devices being archived by the **fast** archiver use **ONLY** the d0ol28
- machine :
**> **setup d0online    on d0ol28 !!!
- > setup chan_archiver
- > Xarr.py  /scratch/smt/psarchive/xxxxxx/dir.200ymmdd-hhmmss &
-       ( xxxxxx can be either  ‘psarchive’ or ‘old’, and y,mm,dd,hh,mm,ss stand
-         for the year, month, day, hour, minute, and second  when this archive file started)
- --------------------------------------------------------------------------------------
- The control of the display program is quite obvious, but you may need to know
- the name of the device you wish to plot, such lists can be found in this book or
- on the web at
- http://d0server1.fnal.gov/projects/Silicon/www/Archiver/fastlist.htm
- http://d0server1.fnal.gov/projects/Silicon/www/Archiver/slowlist.htm
- for the fast and the slow archiver respectively.

## MCH Walkthrough

- Contact Aurelio Juste for any questions about MCH Walkthroughs:
- 722-0033
- x6565

## D0 Radiation Monitoring

_**Shift Procedure for SMT Radiation Monitoring**_
- 	SMT radiation dose is monitored through two systems: BLMs (Beam Loss Monitors) and Fingers (silicon diode detectors residing inside the SMT). The BLM system is the primary protection system. It generates alarm and abort signals according to preset thresholds. Both the instantaneous and integrated doses are recorded and can be monitored through ACNET interface of the system (E35). One of the integrating devices (D0BRUT) is monitored by MCR for making manual trip decisions for integrated doses. The SMT Finger system provides detailed dose monitoring through EPICS and can be used to analyze losses in more detail than the BLM system.
_**Startup**_
- Start E35 ACNET page for BLM monitoring (if it's already not running) on the ACNET console.
- Start StripChart for BLM HV:
- from /projects/D0rad/shift
- Linux:onl-> setup d0online
- Linux:onl-> ./blm_bias.sh &;
_**Monitoring**_
- On E35 page, make sure that all the devices are enabled: green EN should show for each device. A LAM state indicates an alarm condition. If this occurs call the experts.
- At the start of shot setup or store, note down the integrated dose values from the E35 page in the logbook. Monitor the change in these values and inform the shift captain if any of the values reaches the manual abort threshold (shot setup: 2krad, store: 3krad). Shift captain would then request MCR to abort the beam. _[D0-OP_SMT-010](http://d0server1.fnal.gov/Projects/UpgradeProject/Run_II_Procedures/D0-OP-SMT-010.pdf)_ (D0 Silicon Detector Radiation Protection) gives instructions on beam alarm and aborts.
- If BLM HV goes out of bound (<1200V or >2200V) call the expert.
_**Experts**_
- Syed Naeem Ahmed		Ronald Lipton  x4250			 x4132 (630)898-7936     (home)		 (630)266-8242     (pager) (630)788-6940     (work) 		lipton@fnal.gov naeema@fnal.gov

## Paging

- 	When you have trouble with the detector and
- don't know what to do, the first thing you should do is
- page the SMT On-call Expert.
- To page the SMT On-call Expert
- you have to:
- Dial: 9 218 8764
- Wait for a signal
- Enter: 8800#
- You will hear another signal - that is it!  Hang up and wait for the On-call expert to call you back.
- Other Pager numbers:
- Breese Quinn: 266-0565
- Michael Weber: 266-0561
- Eric Kajfasz: 218-9774
- Harald Fox: 847-414-4044
- Petros Rapidis: 905-4069
- RADMON On-call expert: 218-9437