Suggestions for later versions should be sent to S. Avvakumov or R.
Plunkett

Table of Contents

What is the purpose of this document? 3

How do I prepare to take shifts? 3

Fermilab Control Room 3

Where? 3

When? 3

What? 4

Computers, accounts, networking etc 4

Computers in the control room 5

Accounts 5

Printing 6

Shifter\'s Laptops 6

Minos Control Room Logbook 6

Searching CRL 7

Making new entries 7

Minos Software Control Tools 8

Starting/Stopping Minos Control and Monitoring Programs 8

Helper GUI 8

Run Control 9

Run Control GUI 10

Controlling Runs and Run Sequences 11

Far Detector Run Sequence 11

Near Detector Run Sequence 11

Starting Runs and Sequences 11

Reset 12

Abort Run 12

Online Monitoring for Near and Far Detectors 12

Online Monitoring Archive and OmHistory 13

Online Monitoring Gallery 14

Status Web pages 15

Data Acquisition Status Webpages 15

Detector Control System Status Webpage 16

Timing System Status Webpage 17

Summary Status Window 18

Detector Control Systems 19

Far Detector 19

High Voltage 19

Magnet Control 19

Environmental Monitoring 20

Near Detector 20

Event Display 21

Using the JIRA problem tracking system 22

Shifter\'s ToDo List 22

Shifter\'s Checklist Forms 23

What to do when something goes wrong 24

DAQ problems 24

DCS, Network 24

Some Important Numbers 24

Glossary (thanks to Jeff Hartnell) 24

###   {#section .unnumbered}

### What is the purpose of this document?

This is a guide to taking shifts in the MINOS control room at Fermilab;
it is recommended that all first time shifters read it. It contains a
brief description of the control room and the software tools as well as
pointers to more detailed manuals for each of them. This document also
describes shifter\'s responsibilities and what to do if something goes
wrong.

#### How do I prepare to take shifts?

Please make sure you have a Control Room Logbook account before showing
up to take shifts! If you do not have an account already PLEASE request
it at

<http://www-numi.fnal.gov/minwork/computing/crl_accreq.html>.

If you are not sure -- go to the main CRL web page at
<http://www-minoscrl.fnal.gov/minos/Index.jsp> and try logging in.

In order to have a much easier time on your first shift, it is very
useful to read (or at least browse through) the following documents:

-   <http://minos-om.fnal.gov/controlroom/MINOS_Intro.doc> -- MINOS Far
    Detector tutorial, especially useful if you haven\'t been to Soudan
    and have not seen the detector itself;

-   <http://www-numi.fnal.gov/numinotes/restricted/html/numi0900/numi0900.html> -
    introduction to the data acquisition system at the far detector with
    instructions on how to run the system. All major components of the
    data acquisition system and different run types are described in
    this guide;

-   <http://www-numi.fnal.gov/numinotes/restricted/pdf/numi0903/numi0903.html>
    -- online monitoring user\'s guide, it describes how to use the OM
    histogram browser to monitor and debug the data collection process;

-   <http://minos-crl.minos-soudan.org/crlweb/Help/CRLW.htm> -- help
    page for the web interface to Minos Control Room Logbook
    application, describes how to use the CRL -- how find recent
    entries, search, add new text entries, attach plots, etc.

### Fermilab Control Room

#### Where?

The MINOS Control room is on the 12^th^ floor of Wilson Hall, in the
northwest corner room.

Most NuMI-MINOS offices are also on the 12^th^ floor. The NuMI
administrative staff is on the 12^th^ floor crossover and can help with
most administrative questions during normal working hours.

There are four phone lines in the Control Room:

-   (630)-840-3368

-   (630)-840-8751

-   (630)-840-2482

-   (630)-840-6913

#### When?

The shift load will be carried out concurrently by the shift physicists
at Fermilab and mine crew/shift physicists at Soudan. At Soudan, shifts
run weekdays from 8:00am to 5:00pm. At the Fermilab control room there
will be three shifts per day, seven days a week, once routine data
taking is underway. Shifts will run from 8 AM-4 PM, 4PM-12 midnight, and
12 midnight-8 AM. Shifters are urged to come early to coordinate with
the previous shifter.

A credit for one shift at the Fermilab Control Room is given for a one
week long shift starting on Monday and ending on Sunday, or starting on
Wednesday and ending on Tuesday.

First time shifters are strongly advised to come at least a day early
and spend some time at the control room with the previous shifter.

#### What?

On shift you monitor three systems: the Data Acquisition (DAQ), the
Online Monitoring (OM) and the Detector Control System (DCS). You will
also fill out the shift checklist, troubleshoot detector hardware (and
many other) problems, record all new problems in the CRL (online
logbook) and in the JIRA problem tracking system, enter other shift
related information into the CRL, and communicate with the shift takers
at Soudan. Specifics of these activities are in the following sections.
If problems arise and you can't find the answers in the documentation,
then you may have to contact an expert, or the MINOS Run Coordinator.

### Computers, accounts, networking etc

There are currently four computers running FermiLinux LTS 3.03 with KDE
window environment at the Minos Control Room as shown in Illustration 1.
It is important (and easy to achieve) that every shifter became familiar
with this environment and capable of doing all the basic operations such
as opening terminal windows, starting/stopping programs using the
graphical menu interface, moving/resizing windows, switching between
virtual desktops etc. KDE is generally very user friendly and has an
extensive help system. Also many of the MINOS collaborators on the 12th
floor use KDE as their primary working environment and will be able to
help. ![](media/image1.png){width="7.327083333333333in"
height="2.9229166666666666in"}

There is also a PC running Windows 2000 and connected to the NuMI Server
that can be used for various tasks that require Windows software. (One
needs a FNAL-wide Windows account to use this system.)

#### Computers in the control room 

-   Run Control: minos-rc.fnal.gov -- runs data acquisition control
    programs for both Near and Far detectors;

-   Online Monitoring: minos-om.fnal.gov runs online monitoring programs
    for both Near and Far detectors;

-   Beam Monitoring: minos-acnet.fnal.gov -- runs the ACNET accelerator
    console program and Minos Beam Monitoring software tools;

-   Event Display: minos-evd.fnal.gov -- runs the Far and Near detector
    event displays;

-   Near Detector DCS (Windows 2000): runs the Magnet Control and
    Environment Monitoring programs for the Near detector;

All the control PC\'s except the Near DCS are running with more than one
monitor per system. The run control PC is equipped with two monitors
that act as one large screen. The Online Monitoring and the Event
Display PC\'s have three monitors attached each and the Beam Monitoring
computer has four screens. The schematic layout of computers in the
Minos control room is shown in Illustration 2.

#### ![](media/image2.png){width="7.320833333333334in" height="3.341666666666667in"}

#### Accounts

There are no personal user accounts available -- shift takers share a
common account \'minos\' and a common setup. Remote logins for this user
are not allowed -- it's strictly console access. Usually the computers
are already all logged on, but in case they are rebooted or just
accidently logged out you will see a login screen prompting you to enter
username/password. Username is \'minos\'. The password can be obtained
from the MINOS Run Coordinator.

***Networking***

All computers are connected to the internet and allow outgoing network
connections, so the shifters can ssh to their favorite system at
Fermilab or other institutions (e.g. to access their email) using their
password or crypto card. Per Fermilab site computing policy, no incoming
connections except kerberized ssh are allowed.

Sometimes it will be necessary to connect to the remote Minos systems at
Soudan or Near Detector Hall that are protected from the hostile world
with gateway computers. In order to connect to them a project kerberos
principle has been provided for the minos account. This principle can be
initialized with a command:

\[minos@minos-om minos\]\$ minos-kinit

Once the project principle has been initialized, it is possible to ssh
to gateway computers at Soudan and the Near Detector Hall with:

\[minos@minos-om minos\]\$ ssh minos-gateway.minos-soudan.org

and

\[minos@minos-om minos\]\$ ssh minos-gateway-nd.fnal.gov

It also becomes possible to ssh or scp files from one control room PC to
another.

#### Printing

There is a "dedicated" HP LaserJet 4200 printer in the control room. To
print a file you should do one of the following:

-   \$lpr filename -- print a file from a command line; sufficient in
    99% of situations

-   \$kprinter filename -- print a file using KPrinter graphical
    interface; you can select various printing options such as printer
    resolution, print quality, single or dual side mode etc.

-   start KPrinter from the KDE application menu; select a file you want
    to print using graphical interface and print it.

The Mozilla web browser is set up to send print output to the same
printer, so you can print out a webpage, by selecting Print from the
File menu (or just press \'Ctrl P\').

#### Shifter\'s Laptops

Shifter\'s laptops should be set up on the desk to the right of the Beam
Monitoring PC or on the desk near the file cabinet. The laptops have to
be registered with Fermilab Computing to be allowed to connect to the
Fermilab network. Registered laptops can use either the wireless network
or the wired ethernet port located to the right of the Beam Monitoring
PC.

### Minos Control Room Logbook

During your shift, you should record anything noteworthy, any changes to
detector hardware, and fill out the shift checklists. These are all done
via the web interface to Control Room Logbook (CRL) program. A reference
for CRL is at http://www-cpd.fnal.gov/CRL/crlwhelp/CRLW.htm. The web
interface is admittedly quirky and takes some time to get used to, so
try making some "practice" entries first and also try different
searches. Be sure to log in to CRL when making entries so that they are
not anonymous.

#### Searching CRL

  --------------------------------------------------------------------------------------------------------
  ![](media/image3.png){width="3.657638888888889in"   ![](media/image4.png){width="3.6569444444444446in"
  height="3.8986111111111112in"}                      height="3.279166666666667in"}
  --------------------------------------------------- ----------------------------------------------------

  --------------------------------------------------------------------------------------------------------

The CRL starting page has some predefined searches for users\'
convenience as shown in Illustration 3. For example clicking on "Far
Manual Entries, 24hrs" produces a selection of manual (which means
\'made by a person\', to distinguish it from the automatic run entries
made by the DAQ processes) entries made in the last 24 hours and
submitted under Far Detector Operation/Data Collection/DAQ categories.

In order to perform a more sophisticated search, click on the "Search"
button in the top right part of the main CRL page. The search page as
shown in Illustration 4 is organized as a collection of search criteria
to be selected by the user. The \'entry time\' cut is set in the left
side of the search page. Additional selection criteria can be defined in
the right side of the page. The actual search is performed by clicking
on the "Submit" button corresponding to the chosen \'entry time\' cut. A
word search can take a fairly long time so be patient.

#### Making new entries

In order to make an entry in the CRL:

1.  Click the "New Entry" button in the top right part of the page.

2.  Enter your username/password (this page appears only in case you
    haven\'t logged into CRL earlier).

3.  Select the type of your entry. Shifters should select "Shift
    Operations" category once it is enabled. Currently most entries
    concerning control room operations should use "Near Detector/Near
    Detector Operations".

4.  Make a logbook entry. Type your entry in the text window in the top
    left part of the page. If you want to fill out one of the forms,
    select the form name from the list to the right of the text window,
    and then click "Insert Form". Fill out the form and click "Submit".

5.  Save your entry by clicking "Save".

The process is shown in Illustration 5 and Illustration 6.

  --------------------------------------------------------------------------------------------------------
  ![](media/image5.png){width="3.6555555555555554in"   ![](media/image6.png){width="3.660416666666667in"
  height="3.088888888888889in"}                        height="2.9208333333333334in"}
  ---------------------------------------------------- ---------------------------------------------------

  --------------------------------------------------------------------------------------------------------

### Minos Software Control Tools

There are a number of tools available to the MINOS shifter that are
designed to monitor and control various aspects of the data collection
process. Here we describe all the control programs and the status web
pages that have to be monitored during the shift.

### Starting/Stopping Minos Control and Monitoring Programs

#### Helper GUI

In order to run the run control program, the online monitoring or the
event display from the Fermilab Control Room, an ssh tunnel to Soudan or
to the Near Detector Hall has to be established first (one ssh tunnel
for each program is needed). A simple helper GUI, shown in Illustration
7, makes it easy for the shifters to start/stop/restart all control
programs. For example to start the run control GUI:

1.  start the helper GUI by clicking on the desktop button "Run Control
    \[Soudan\]" located in the top left corner of minos-rc.fnal.gov
    screen

2.  start the ssh tunnel by clicking Start button in the top part of the
    helper GUI, wait several seconds until SSH Tunnel Status shows
    "Running"

3.  start the run control GUI by clicking Start button in the bottom
    part of the helper GUI, wait several seconds run control window will
    appear

The process of starting ND run control, ND and FD Online Monitoring, ND
and FD event displays is exactly the same -- start the corresponding
helper GUI -\> start ssh tunnel -\> start program.

The desktop icons for starting the ND/FD Online Monitoring programs are
located in the top left corner of the minos-om.fnal.gov screen; the
icons for starting the event displays are in the top left corner of the
minos-evd.fnal.gov (the screen labeled \'EVD Control\' in Illustration
2).

It can happen that the run control, the OM or the event display starts
acting abnormally, hangs or crashes. In all these cases it is easy to
restart the processes. If the program is still "alive", clicking the
\'Stop\' button in the bottom part of the helper GUI will first try to
stop the process gracefully, and then will kill the process five seconds
later. Then if ssh tunnel got disconnected (SSH Tunnel Status says
Stopped), repeat steps 2-3. If ssh connection is still active, then step
3 is sufficient. ![](media/image7.png){width="3.6666666666666665in"
height="2.6013888888888888in"}

It is not necessary and not recommended to start several helper GUI
programs for the same task simultaneously. Before clicking on the
desktop icon, please check the process on the screen to see if another
helper GUI is already running.

### Run Control

The single most important shifter\'s responsibility is to control the
data collection process using the Run Control program. During your
shift, you should make sure the DAQ is running, or at least understand
why it isn't (e.g., it may be turned off for detector maintenance,
software upgrade, or some other reason).

Run control allows you to start/stop runs and run sequences, choose run
conditions and run types, do more expert tasks such as disable/enable
readout branches, reset DAQ components etc. The detailed information on
its operation can be found in the DAQ user\'s guide, available at

<http://www-numi.fnal.gov/numinotes/restricted/html/numi0900/numi0900.html>.

Under most circumstances Fermilab control room shifters are not allowed
to perform any DAQ expert tasks. They only have to deal with a very
small subset of run control functionality.

![](media/image8.png){width="5.879166666666666in"
height="5.405555555555556in"}

#### Run Control GUI 

The run control GUI can be run on many PCs but only one GUI can be in
control at any given time. A small user window appears next to the main
run control GUI window that shows a list of users currently running an
instance of the run control, an \'x\' mark next to the user name
indicates the user who has control. The appearance of the GUI depends on
whether you have control of the detector or are just watching.

The run control GUI has a lot of information on it. There are four or
five main sections depending on whether you have control or not:

1.  Small boxes describing the runType, number of time frames, run
    duration, etc. Look here if you want to know what the DAQ's doing.

2.  The pale blue control buttons (these disappear if you don't have
    control of the detector). They are intuitively labelled: Start run,
    stop run, etc.

3.  A number of buttons corresponding to different parts of the DAQ
    (e.g. BRPs, FPC, TPC, etc) that change colour depending on the state
    of that part of the DAQ (green is good, red is bad, yellow means
    \'idle\'). When these buttons are clicked a little window pops up
    that allows you to talk to the processes running on that particular
    PC. The options available include "connect" and "reset" which can be
    used in the event of a problem.

4.  Various displays showing the data rates in various parts of the
    detector. These are instructive to watch.

5.  The message logger. This lists all the messages that are output by
    the various DAQ components. It can be configured to list messages of
    a certain priority (e.g. only warnings, all messages, or various
    intermediate levels). It can be interesting to watch this to get a
    feel for how things work.

#### Controlling Runs and Run Sequences![](media/image9.png){width="2.2319444444444443in" height="1.5069444444444444in"}

There are a number of different ways to control the DAQ. Runs can be
started and stopped manually or RC can control them. Some runs will stop
themselves when they have finished and the DAQ will then just wait for
new commands. Alternatively run sequences can be used which are
programmed to stop and start different runs after given durations. A run
sequence can be programmed to loop over a sequence forever or finish
when it gets to the end of the last run in the sequence. Furthermore, a
particular run can be set to start at a given time, e.g. do a charge
injection calibration run every Saturday at 3 PM.

##### Far Detector Run Sequence

The only run type (run sequence actually) Fermilab shifters have to deal
with is the "RS24Hr" sequence that is our default cosmic, atmospheric
neutrino data taking mode. This sequence programs the run control to
loop over the following run types:

1.  VADrift run -- 67 second long charge injection run

2.  Pedestal run -- 30 seconds

3.  Physics run -- 4/5 planes triggered with interspersed LI -- 8 hour
    long, gets split into four subruns

##### Near Detector Run Sequence

The Near Detector run sequence will be implemented later based on the
operational experience. Currently Near Detector runs are controlled
manually.

##### Starting Runs and Sequences

In order to take data you have to be in control of the Far/Near
detector. If you aren't in control, then you can gain it by clicking in
the Control menu at the top of the window, near the center. There are
several options available on that menu depending on where the DAQ was
last used. If no one has control, then click "take control". If someone
else has control, then you can "request control" and wait for a response
from the current user. Control is usually transferred from Soudan to
Fermilab at the end of day shift at Soudan. TAKING CONTROL SHOULD ALWAYS
BE ENTERED IN THE CRL.

Once you have control you are in a position to start taking data.

Click on the blue \"Start_Run\" button. You should see the "Select Run
Parameter" window pop up. There are several sections in this window. You
will have to enter data in three sections, which are described below:

1.  "Select Run Type": There are many run types and run sequences (a
    list of run types of different durations). The run types are
    described in the \"Run Types and Run Sequences\" section below.

2.  "Select Run Information": Hit the "default" button, fill in your
    name and change any of the other fields appropriately.

3.  "Select Run Comment": Put a comment that is specific to the run or
    run sequence.

4.  Click "OK" and the buttons should go green. If the GUI fails to
    start taking data (the buttons stay yellow or go red) then follow
    the procedure in the \"If you get a GLOBAL error on the Run Control
    GUI\'\'.

When clicking "Stop_Run" during a run sequence you will be presented
with a window where you can select to stop the current run but continue
with the sequence or just stop completely.

##### Reset

When you get a GLOBAL error on the GUI, the DAQ should be recovered by
clicking "Reset". This will return all the processes to the 'idle'
state. Once the buttons on the GUI are showing 'idle' (yellow), then a
new run can be started. Note that there is a rare case where a running
process will go to error when you click reset. If this happens, then you
should click reset a second time.

If the global state does not return to idle, then a problem persists
that will require further attention. Have a look at the messages on the
GUI to see what the problem is.

If one of the processes has not returned to the idle state and has
remained in error, then it can be communicated with directly using the
GUI. See the section "If you want to reset/communicate with a particular
process".

Note: Clicking "Abort_Run" won't stop a run that is in the 'error'
state. It can only stop a run that is in the 'running' state. If
"Abort_Run" is clicked when the global state is 'error', then the GUI
will display a message saying that this can't be done and nothing else
will happen.

##### Abort Run

"Abort Run" can only be executed for running processes. For processes in
error the "Reset" command must be used. Abort run should only be used
very rarely as it could cause data loss and some processes will revert
to their default state thereby forgetting what they have done. In almost
all cases you should simply use "Stop Run" to return running processes
to the idle state.

### Online Monitoring for Near and Far Detectors

The Minos Online Monitoring program allows the shifter to closely
monitor the data collection processes in each detector by filling out
many histograms related to all aspects of data taking, raw electronics
data, charge and light injection data, channel occupancy etc, etc, etc.
To view the online histograms a graphical viewer program should be
started on the minos-om PC by clicking on the icons provided on the
desktop. There are should be two Online Monitoring displays running, one
for each detector. When the viewer window appears, click on the "Open"
button to open a connection to the server program and wait until you see
the top level histogram folder "Producer" in the bottom part of the
window (usually \~2-5 s). You can navigate over the histogram folder
tree using a mouse -- double click on the folder to "expand" it (i.e.
see its contents), double click on a histogram name to have it appear in
a canvas window. Detailed information on how to use the OM viewer
program and what various histograms mean can be found at

<http://www-numi.fnal.gov/numinotes/restricted/pdf/numi0903/numi0903.pdf>.

It\'s possible to save any OM plot in order to print it or attach it to
a CRL or JIRA entry. Click on "File" menu of the window with the plot
you want to save, then choose "Save As", select file type "gif", give it
a meaningful name and save it in /home/minos/plots directory.

![](media/image10.png){width="4.289583333333334in"
height="5.413194444444445in"}

#### Online Monitoring Archive and OmHistory

The Online Monitoring program resets all histograms to zero at the
beginning of every new physics run and it takes at least two hours to
accumulate enough data to make it worth looking at the online plots. In
such situation it is recommended to look at the last archived OM file.
Clicking on the desktop icon "OM archive Far Detector" opens a folder
with archived OM files select the last available physics run and click
on the icon. This procedure opens a standard ROOT histogram browser.
Click on "ROOT files"-\>File Name in the left part of the ROOT browser
window and you will see the list of histogram folders. All checklist
histograms can be found in the "Checklist plots" folder.

  ----------------------------------------------------------------------------------------------------------
  ![](media/image11.png){width="3.5805555555555557in"   ![](media/image12.png){width="3.582638888888889in"
  height="2.5694444444444446in"}                        height="2.365972222222222in"}
  ----------------------------------------------------- ----------------------------------------------------

  ----------------------------------------------------------------------------------------------------------

#### Online Monitoring Gallery

The OM Gallery is a collection of sample OM checklist plots with
comments, which is available at
<http://minos-om.fnal.gov/cgi-bin/photo/index.cgi>. Shifters are
encouraged to browse through these plots and add their
comments/questions. They are encouraged even more to add sample plots to
this collection by first saving the plots as \'gif\' files and then
uploading them to the OM Gallery using the web interface at
<http://minos-om.fnal.gov/cgi-bin/photo/protected/manage.cgi?mode=manage>.

  -----------------------------------------------------------------------------------------------------------
  ![](media/image13.png){width="3.6569444444444446in"   ![](media/image14.png){width="3.6569444444444446in"
  height="3.536111111111111in"}                         height="3.0791666666666666in"}
  ----------------------------------------------------- -----------------------------------------------------

  -----------------------------------------------------------------------------------------------------------

### Status Web pages

The status web pages are extremely useful for quick, convenient
monitoring of what's going on with data collection. They are available
to the MINOS collaboration, but it is still the primary responsibility
of the person on shift to check them often and react to any
abnormalities.

You can reach all the status web pages by following links from the Minos
Control Room web page at
<http://www-numi.fnal.gov/internal/controlroom/index.html> (this page is
set as the home page and it loads automatically when the Mozilla browser
is started).

After the initial loading of the status web pages in the Mozilla web
browser, they update automatically every minute to reflect the status of
the system. It's necessary to pay attention to the time stamps at the
top of the pages. If a time stamp shows that the last update happened
more than five minutes ago, the page has become stale, which indicates a
possible problem. In case manual reloading of the page doesn't solve the
problem, experts have to be notified.

#### Data Acquisition

The DAQ webpages are located at
<http://minos-crl.minos-soudan.org/cgi-bin-daq/daq_status.cgi> for the
Far detector and at <http://minos-om.fnal.gov/cgi-bin/daq_status.cgi>
for the Near detector. They are shown in Illustration 15 and
Illustration 16 respectively. The status pages contain the following
information:

-   Top Section

    -   Global state of the DAQ system

    -   Run Number and Type

    -   When the run was started and its duration (as of the last
        update)

    -   Number of time frames (equal to the number of seconds since run
        has been started) and number of snarls recorded

    -   Number of dropped time frames

    -   Data disk usage -- shows fractional disk usage. A high number
        \>50% indicates a possible problem with the data transfer from
        Soudan or the Near Detector Hall to Fermilab and potential disk
        overfill -- experts have to be notified.

    -   ROPs (=Read Out Processors) enabled -- in normal running all 16
        ROPs should be enabled

    -   Trigger Mask

    -   DCS status

    -   LI status -- shows if the interspersed LI data is being taken
        during a run. If LI is enabled it will indicate which pulser box
        is active.

    -   Run Comment -- comment string, entered by the DAQ operator (that
        might have been you!)

-   Bottom section

    -   Singles rates read from each of 16 crates. Normal rates
        correspond to green bars; higher rates to yellow or red bars.
        High rates in a crate usually correspond to LI activity; check
        the LI to see whether the pulser box for the high rate crate is
        active.

    -   Data rates in the DAQ system -- input rates in BRP's 0-5 (Branch
        Readout Processors), TP's 0-3 (Trigger Processors)

    -   Output data rate in snarls/s and kB/s written to disk

  ----------------------------------------------------------------------------------------------------------
  ![](media/image15.png){width="3.6597222222222223in"   ![](media/image16.png){width="3.657638888888889in"
  height="4.2444444444444445in"}                        height="4.189583333333333in"}
  ----------------------------------------------------- ----------------------------------------------------

  ----------------------------------------------------------------------------------------------------------

Note that the shifters should be running the FD/ND run control programs
during all normal operations. The run control programs provide all the
necessary information about the running conditions.

#### Detector Control System

DCS status webpages are located at
<http://minos-crl.minos-soudan.org/webdcs/dcs_status.html> for the Far
detector and at <http://minos-om.fnal.gov/webdcs/dcs_status.html> for
the Near detector, shown in Illustration 17 and Illustration 18
respectively.

They display the following information:

-   Global DCS status

-   Status of HV (High Voltage) systems

-   State of the RPS (Rack Protection System) software

-   Status of the Environment Monitoring software

-   Status of the Magnet Control and Monitoring software

-   Status of the Chiller monitor (Far detector only)

-   Pump Status (Near detector only)

Monitoring the DCS status webpages is one of the main shifters\'
responsibilities. Both pages should be opened in separate Mozilla
windows and be checked regularly during the shift.

  ----------------------------------------------------------------------------------------------------------
  ![](media/image17.png){width="3.658333333333333in"   ![](media/image18.png){width="3.6569444444444446in"
  height="4.53125in"}                                  height="4.761805555555555in"}
  ---------------------------------------------------- -----------------------------------------------------

  ----------------------------------------------------------------------------------------------------------

In case this page becomes stale (and can not be fixed by manual
reloading) or any of the background colors on the page turns red,
experts should be notified as soon as possible.

#### Timing System

-   TimeMaster status: the top lines show the program's Run Control
    state, and whether or not a Run Control process has attached to the
    TimeMaster running. The time shown in the upper right hand corner is
    the TPC time. The master clock time is used for the entire
    experiment. If there is an error, it is shown in this section. Error
    codes are mostly for expert use. Consult the log file to see more
    descriptive comments. The Run Time is an estimate and should not be
    used. The "Time Left" field is used only when TimeMaster is in
    "control" of the length of the run: this is used for tasks such as
    pedestals or charge injection runs.

-   Far Hardware status: next is shown the status of the TCU. This is
    updated constantly. These items should be self-explanatory. The
    "GPS-\>TCU" field should always be "synched". If the TCU becomes
    un-synched (i.e. a problem with the GPS) the system will be driven
    into error.

-   Near Hardware status: this is a placeholder for the mode where
    TimeMaster is running at the Near Detector. The ND system is
    different and documented elsewhere.

-   Current Run Status: next is the current run status. This shows run
    number, etc, and shows how many execute bursts are still in the
    queue to be sent out to the FD hardware.

-   Pending Run Configuration: the Pending Run Configuration does not
    show the state of the hardware, but rather shows how the most recent
    Run Prepare sequence wants to configure the run.

-   TimeGoblin Status: at the bottom is the most recent status report
    from either the NTP process or the TimeGoblin process that is
    controlling the system clock.

  ----------------------------------------------------------------------------------------------------------
  ![](media/image19.png){width="3.6590277777777778in"   ![](media/image20.png){width="3.658333333333333in"
  height="4.851388888888889in"}                         height="4.897222222222222in"}
  ----------------------------------------------------- ----------------------------------------------------

  ----------------------------------------------------------------------------------------------------------

#### Summary

A link is available from each of the DAQ status pages that brings up a
DAQ+DCS summary window. This window displays four boxes. One shows the
status of the FD DAQ, one shows the status of the FD DCS, another shows
the ND DAQ status, while the last shows the status of the ND DCS. If any
of these systems go into error, an audible alarm will sound.

![](media/image21.png){width="2.95625in" height="1.45625in"}

### Detector Control Systems

Detector Control System (DCS) is an ensemble of environmental monitoring
subsystems which are intended to continuously view MINOS Far and Near
detectors during runtime operations. It is the essential mission of DCS
to provide prompt warning flags for errant run conditions. DCS
functionality is provided by multiple hardware subsystems and their
control and monitoring software.

#### Far Detector

You can check the status of the control systems and control the voltage
to the photo tubes using the Detector Control System (DCS). The DCS is
run on a Windows PC at Soudan that can be remotely controlled using VNC
connection. The session can be started using the helper GUI. After you
start the GUI, ssh tunnel and the VNC connection, you will see the
screen of the Windows computer. You can control the DCS computer just as
if you were physically at Soudan. Beware that if this VNC connection is
started during normal working hours when the crew and the shift
physicists are at the mine at Soudan, you may interfere with them
working with the DCS controls, because VNC works as if the PC had two
mice and keyboards connected. Therefore, it\'s strongly recommended that
you use the VNC connection only as a passive viewer until after 5:30pm.

There are three main elements to the DCS: High Voltage Control, Magnet
Control, and Environmental
Monitoring.![](media/image22.png){width="7.327083333333333in"
height="2.9229166666666666in"}

##### High Voltage

The DCS interfaces to two computers, each of which control the HV
mainframes to one supermodule. These computers, dcshv1.minos-soudan.org
and dcshv2.minos-soudan.org, are both linux machines. Documentation on
the HV system is at

<http://www-numi.fnal.gov/minwork/fardet/documentation/hv.pdf>.

##### Magnet Control 

The DCS magnet operation window displays magnet currents and
thermocouple readings for both supermodules. Ordinarily, you shouldn't
have to do anything with the magnet remotely unless something is very,
very wrong. Magnet control documentation is at

<http://www-numi.fnal.gov/numinotes/restricted/pdf/numi0870/numi0870.pdf>.

##### Environmental Monitoring

The environmental monitoring window displays the temperature, humidity,
barometric pressure, and radon levels in the detector hall.
Documentation is at

<http://www-numi.fnal.gov/numinotes/restricted/pdf/numi0869/numi0869.pdf>.

#### Near Detector

The Environmental Monitoring and Magnet Control programs for the Near
detector are running on the Windows 2000 PC in the Minos control room
near the Beam Monitoring PC. A screen shot is shown in Illustration 23.
Their use and functionality are covered by the documentation listed
above.

![](media/image23.png){width="6.9375in" height="5.197916666666667in"}

Illustration 23 Near Detector DCS System

### Event Display

The event display program allows you to look at the real time events
being recorded at the Minos Far and Near detectors. These displays
contain information on detector tracking capabilities and also detector
performance. ![](media/image24.png){width="7.31875in"
height="6.152083333333334in"}

Start the event display using the user control GUI in the same way as
run control or online monitoring: click on the desktop button "Event
Display \[Soudan\]" -\> start ssh tunnel -\> start event display. The
event diplay window will appear, but the event show does not start
automatically -- one has to click on the "Next" or "NextPass" button to
display the next available event. The difference between the two buttons
is that "NextPass" button displays only events that pass a simple noise
filter, requiring \>3000 ADC in the event, while "Next" displays the
first event it receives. Be patient when you first click "Next" or
"NextPass" after starting the event display because it takes up to 3-4
minutes to initialize all databases needed by loon. It is possible to
start the continious event processing/display by clicking the
"FreeRunning" button and then the "Next" or "NextPass" button. To stop
the automatic running, click on "StopRunning". A more detailed event
display tutorial is at <http://minos-om.fnal.gov/evd/>.

### Using the JIRA problem tracking system

In order to facilitate the process of tracking problems, a web-based
database system called JIRA is used (origin of the name: this project
was started as an alternative to Bugzilla which sounds like Godzilla or
Gojira in Japanese and the name got shortened to JIRA). The JIRA entry
webpage is at
[http://minos-om.fnal.gov:8080](http://minos-om.fnal.gov:8080/). In
order to start using it, shifters have to register first by getting a
username and password. After getting registered and logging in, one can
browse all the \'registered\' problems. Any new problems noticed during
the shift should be reported in the CRL and also submitted in JIRA. In
order to submit a new problem to the JIRA database, click on the "Create
New Issue" button and enter the problem description and severity. If you
have a suggestion how to fix the problem, please enter it too.

  ----------------------------------------------------------------------------------------------------------
  ![](media/image25.png){width="3.6569444444444446in"   ![](media/image26.png){width="3.658333333333333in"
  height="3.7604166666666665in"}                        height="3.377083333333333in"}
  ----------------------------------------------------- ----------------------------------------------------

  ----------------------------------------------------------------------------------------------------------

### Shifter\'s ToDo List

There are several main goals for the regular shift taking:

-   ensure the highest uptime of both Far and Near detectors

-   diagnose and fix problems quickly and efficiently

-   alert the system experts and work with them in case of problems

-   perform regular inspections of the data taken, monitor detector
    systems and environment conditions by filling out shifter\'s
    checklists

The approximate shifter\'s ToDo list:

1.  Be at the control room early. Arrive 30 minutes before your official
    shift start time.

2.  Carefully read all CRL entries entered in the last 24 hours (if
    it\'s your first day on shift, then browse thorugh last several
    days, paying attention to all special tasks accomplished, problems
    discovered and fixes performed).

3.  Begin filling Start_Shift.htm form in the Control Room Logbook.

4.  Restart both run control GUIs on minos-rc.fnal.gov screen. A window
    will appear asking for the current operator\'s name. Type your name
    in the appropriate box. Restarting the RC GUI ensures that you are
    identified as the current shifter throughout your shift.

5.  Find out who are the shift physicists and crew supervisor. The shift
    schedule is posted at
    <http://www-numi.fnal.gov/internal/fardet/shift_index.html>

6.  Talk to either the previous shift physicist or a crew member and
    discuss what happened during their shift. Note any new problems
    found or old ones fixed. Inquire whether there were any hardware
    changes, special activities etc. Ask if there are any requests for
    special data taking and remind them to release DAQ control before
    they leave. Then wish them a safe drive home.

7.  Take control of the DAQ using the run control GUI.

8.  Proceed with filling out shifter\'s checklists (described in the
    next section); try to find and diagnose hardware problems; record
    your findings in the online logbook.

9.  Regularly check DAQ, DCS status webpages (keep both of them open on
    top of one of the multiple control room screens).

10. Spend time looking at the real time events using the event display.

11. Browse through Online Monitoring plots, not just the ones from the
    checklist.

12. Record anything happening in the online logbook.

13. Enter all new problems into the JIRA issue tracking system.

14. At the end of the shift, fill out End_Shift.htm form. Write a shift
    summary, list all the problems discovered and suggestions on how
    they should be fixed.

### Shifter\'s Checklist Forms

There are two sets of forms, one set for the Near Detector, one for the
Far Detector. Each set consists of three checklist forms, one for DCS
status, and two for Online Monitoring.

Filling out the DCS checklist is fairly straightforward -- all the
necessary information can be found on the DCS status webpages. It\'s a
little boring, but remember that the point of this form is not to save
the values of temperature, pressure, magnet currents etc, but rather to
ensure that these values are being looked at regularly by the shift
person. It\'s much easier to stop/prevent any problems if they are
noticed early. For example, if the chiller malfunctions, the temperature
of the magnet cooling water starts to rise and eventually it will
trigger software and hardware alarms that will shutdown the coil. But
the temperature rise is a fairly slow process (several hours) and can
easily be detected early by the shifter before it triggers the alarms
and be dealt with immediately.

The Online Monitoring checklist form is subdivided in two parts --
Om1CheckListFar.htm (Om1CheckListNear.htm) and Om2CheckListFar.htm
(Om2CheckListNear.htm). In order to fill them out, the shifter needs to
start the Online Monitoring Histogram Viewer for the appropriate
detector and browse through all plots in the Checklist folder. It\'s
important to let the online statistics accumulate, so make sure that the
data run has been running for at least two hours before proceeding with
the checklist. To answer the question whether a certain OM plot is
"good" or shows problems shifters should consult the OM manual at
<http://www-numi.fnal.gov/numinotes/restricted/pdf/numi0903/numi0903.html>
and browse through the Online Monitoring Gallery at
<http://minos-om.fnal.gov/cgi-bin/photo/index.cgi> -- a collection of
sample OM plots with comments and guides how to find problems.

### What to do when something goes wrong

First, common to all problems encountered, enter all relevant
information into the CRL: how you discovered the problem, how you plan
to fix it, and later enter information of what you\'ve done and what the
results are.

##### DAQ problems

In case of DAQ errors, try resetting it (consult the DAQ guide). Usually
it recovers and switches to the idle state. Then it\'s possible to start
the data taking again. In case of recurring problems or if reset does
not clear all problems, contact Geoff Pearce (call if it\'s reasonable
time in UK, otherwise use email, cc to Bill Miller, Alec Habig, and Rob
Plunkett who must be notified about all such problems, too).

##### DCS, Network 

Most of the DCS and network problems - such as the chiller shutting
down, coil overheating, network down, a data crate has tripped off, RPS
warnings - cannot be fixed remotely. The minecrew should be called in
night or day if there is any problem that stops data taking. The on-call
cell phone number carried by a minecrew member is 218-780-9701. In case
the on-call person doesn\'t answer, call Bill Miller or Jerry Meier.

In case of software problems with the DCS systems such as webpage stops
updating or says that one of the HV, environment, magnet, RPS, chiller
monitors is not running contact Erik Beall ([is it still
Erik?).]{.underline}

There is a list of experts to call in case of problems that shifters
should use to look up the subsystem experts for help. During reasonable
hours, call the experts for most problems. During late night hours, you
should only call someone in case of an emergency.

The list of experts is at

<http://www-numi.fnal.gov/numinotes/restricted/pdf/numi0896/numi0896.pdf>

Instructions on how to contact the experts are in

<http://www-numi.fnal.gov/numinotes/restricted/pdf/numi0897/numi0897.pdf>

At Fermilab, you must dial 9 for an outside line.

### Some Important Numbers

-   At Fermilab, in case of an emergency, call extension 3131.

-   Control Room Numbers: x3368; x8751; x2482; x6913

-   Soudan Mine: 218-753-6611

-   Soudan on-call cell phone: 218-780-9701

-   Computing Help Desk: x 2345

### Glossary (thanks to Jeff Hartnell)

-   ASD: Amplifier Shaper Discriminator -- chip that performs triggering
    in MINOS Far Detector; there is one ASD Lite chip per VFB.

-   BRP: Branch Readout Processor. These PCs control the ROPs that live
    in each of the electronics crates via an optical PVIC interface.

-   Crate: This is used to refer to the VME crate that sits in a rack on
    the mid-level at the mine. There are 16 of these, 8 on each side of
    the detector.

-   CRL: Control Room Logbook. An electronic logbook for record keeping.

-   DAQ: Data AcQuisition.

-   DCP: Data Collection Process/PC. This program/PC collects the data
    from the TPs via Ethernet.

-   DCS: Detector Control System. This covers a wide range of processes
    from temperature monitoring to PMT HV control and RPS.

-   DDS: Data Distribution System. The online processes (e.g. online
    monitoring) get the latest DAQ data from this PC.

-   ETC: Event Timestamp Controller

-   FPC: Flasher PC. This PC controls the pulser boxes that do the light
    injection. The LiMaster process runs on this PC.

-   GLOBAL: This refers to the global state of the DAQ. The global state
    is only good (idle or running say) if the state of all the essential
    components is good (In other words it is an "and" of the states of
    all the essential components).

-   GPS: Global Positioning System. Gives very accurate timing
    information, that's why we use it to synchronise the near and far
    detectors. The time from the GPS is fed out to the TRCs.

-   HV: High Voltage. Used to power the PMTs and give them the high gain
    we need. PMTs typically run at 700-800 volts .

-   LAN: Local Area Network. A simple Ethernet system to allow computers
    to communicate.

-   MBRP: Master Branch Readout Processor

-   NTP: Network Time Protocol

-   OM: Online Monitoring.

-   PCI: Peripheral Component Interconnect. The standard bus for
    connecting devices such as network, sound, video cards, modems to a
    PC.

-   PDU: Power Distribution Unit. Only to be used by experts.

-   PVIC: PCI Vertical InterConnection. This is the name of the link
    that is used to allow high speed communication between the ROPs and
    the BRPs and between the BRPs and TPs.

-   RC: Run Control (PC). This is the program/PC that controls the DAQ.

-   ROP: ReadOut Processor. Each electronics crate contains one ROP.
    These are daisy chained together by an electronic PVIC interface.

-   RPS: Rack Protection System. This is part of the DCS.

-   SM1 and SM2: Super Module 1 and 2. This refers to the physical
    division of the MINOS far detector into two parts, which are
    separately magnetised.

-   Snarl: An event, read out by the DAQ and recorded in a data file. At
    the Far Detector one snarl corresponds to one real event - like a
    cosmic muon or neutrino interaction, at the Near Detector a snarl
    can contain several unrelated events.

-   TCU: Timing Control Unit

-   TimeGoblin: Daemon process that controls the system clock

-   Time Frame (TF) is a DAQ unit of time that is arbitrarily set to one
    second.

-   TRC: Timing Readout Card. The card that keeps all the crates in
    time.

-   TP: Trigger Processor. The main job of these PCs is to divide the
    data up into snarls and decide whether to write each snarl to disk.
    The basic trigger requires N of M planes (e.g. 4/5) in a certain
    timeframe to be hit otherwise the data is thrown away. The TPs are
    connected to the BRPs via an electrical PVIC connection.

-   TPC: Timing PC. This PC controls the timestamps for all the data and
    allows synchronization between the near and far detectors.

-   UDS: Universal Device Server. This device works by encapsulating
    serial communications between itself and the ROP then transporting
    them over ethernet. These devices allow simple remote serial
    communication with the ROPs.

-   VA Chip: The name of the chip that is connected to each M16 PMT.
    (numbered 0-2 on the VFB)

-   VARC: VA Readout Controller. The name of the MINOS VME card that
    sits in the electronics crates. There are three of these cards in a
    crate (numbered 0-2).

-   VFB: VA Front end Board. The electronics board attached to a MUX box
    (numbered 0-1 on the VMM)

-   VME: The VersaModule Eurocard bus is the system used by the VARCs
    and ROPs to talk to each other and pass their data around.

-   VMM: VARC Mezzanine Module, up to 6 of these electronics board sits
    on the VARC (numbered 0-5)