Overview of D0 data processing on farms.    


This note is based on a conversation with Kirill Denisenko and my notes from
recent production meetings.   As much of this was copied from my notes,
directory names are not guaranteed.




D0 Data

When D0 takes data, it is written onto 8mm tapes, however, the essential unit
of data is the file.  When it is produced each file gets a name like:

ALL_099999_01.X_RAW01 

where ALL is the stream, 99999 is the run number and 01 is the file number.
As this file travels through the D0 data processing system the suffix will
change to reflect its status and a time stamp will be added.  

This run  we will write several thousand tapes, each with many files and 
elaborate means  are needed to keep track of the data.  Essentially, at each
phase there is a list of valid files, kept either in a normal ASCII file or in
the Fatmen database.  A file only exists in the eyes of D0 software if it is
physically on a tape or disk AND its name appears correctly in the appropriate
list. Thus whenever a processing step such as reconstruction (RECO) occurs, the 
job is not complete until both the tape or disk file is written and the  name
of the new file is logged in the appropriate database or list.

The Farm
________

A farm is a collection of CPU's which are used to process data from a single
input stream.  In general there will be an input control process, which takes
an event from an input file and doles it out to a CPU (often called a worker
node) , a reconstruction process  on that worker node which reconstructs the
event and an output process which collects the event from the worker node once 
it is processed and writes it to an output file.  All of these processes are
running asynchronously, thus minimizing idle time.  This configuration allows a
small number of IO devices to have access to a very large CPU capability.  One
disadvantage is that, since events take different amounts of time to process,
events will be shuffled on the output tape.

D0 production implementation
____________________________

D0 uses its own Production manager called TPM on the FNAL farms.  This system
has been optimized for the D0 environment in which there are very large events
(500KB) with  reasonably small CPU requirements (18 secs on 30 MIP machine)
per event.

The production manager can control several 'virtual' machines or farms each of
which  has a list of worker nodes assigned to it.  Each virtual machine has
several processes running on it.

The virtual machines have names like fnsfe_1, fnsfe_2 where fnsfe is the real
machine that the control program is running on.  Thus a single physical machine
can actually control several farms of worker nodes. Each virtual machine has
the following processes going.



1) The input spooler - this copies a file from the input tape to a disk area.
(Process names are : inspool_fnsfe_1 where 1 stands for the spooler number not
the machine number.)  Actually the file is read into  a temporary area and
moved into another called $PRODIR/raw when reconstruction processing starts.


2) The input server - this reads an event from the disk file and sends it off
to  a worker  node to be processed.  (The process name is insrv_fnsfe_0)


3) The output server - this receives the event from the worker node and writes
it to an output disk file (Process name is outsrv_fnsfe_0).

4) The output spooler - this writes the output disk file onto tape when it is 
complete.  (outspool_fnsfe_0 where 0 is the machine number)


The script d0reco_prol.csh starts up the data servers and establishes
communications with the worker nodes. 

On each worker node
the process whkshell and  an executable with the reconstruction job is running.
There are also shared memory managers called inreader.x and outwriter.x.

CONTROL
_______


The master control for this whole thing comes from a master process called
tpm_submit_job.  This process controls all of the machines and   wakes up every
800 seconds and checks to see if a job (reconstruction of a tape) has been
completed on any of them.  If not, it goes back to sleep.  If so, it looks for
the next tape in the queue and starts up another job on the empty virtual
machine.  The control for this is set up through specification files:

Specification files for machines, projects ...
_______________________________________________ 

  $PRODIR/resources/resource 

is the master file which contains a list of the following:

Virtual      
Machine     Project         Status    I/O spooler

fnsfe_0     full_d0recoV12  FARMLET   1
fnsfe_1     special_stuff   FARMLET   2
...
...

This says that virtual machine fnsfe_0 should be running the full_d0recoV12 
project with input coming from spooler 1.

Each virtual machine is specified by a file 

  $PRODIR/resources/resource.fnsfe_0

which contains a list of nodes assigned to that  system and their status.  You
can remove broken nodes from the system by  changing this file.

The projects are specified by files like:

  $PRODIR/project/PROJECT.fulld0recoV12
 
which points to the executable and the LIBPROD area. (LIBPROD tells which RCP
constants files to use).

How does it know which tapes to process?
________________________________________

You now have specified a virtual machine with input and output spoolers and
chosen your executable, now how does that machine know which tape to run? At
startup, the production manager looks in the file $PRODIC/resources/ injobs to
see if there are any tapes waiting to be processed.  This file has lines of the
format:

WMXXXX   fulld0recoV12  WAITING     0
WMYYYY   fullD0recoV12  SUBMITTED   1

WMXXXX is the tape number, fulld0recoV12 is the project you want to do to it,
WAITING means that it has not been dealt with yet and the final number shows
the error count.  If ierr>2, the job has been attempted twice and the tape is
considered  to be bad.   The information on the first WAITING tape is sent to
the production manager which finds an empty virtual machine set up to run the
chosen project and starts it up.

The production manager also looks in the file blanks for the next empty 
tape for output.

The production manager loops over the virtual machines every 800 sec to see if
one of the spoolers has finished its tape. If it has, it gets another tape name
from the injobs list  and tells the virtual machine to start processing them.

Output tapes keep going until they are almost full and a new tape is started.
There is not a one-to-one correspondence between input and output tapes.

WHERE DOES INJOBS come from? 
____________________________

$PRODIR/history/itapes_moved contains a list of vaulted tapes and the files on
them.  Every hour a process checks to see if the file has been updated.  If it
has,  tapes in that list which have not been processed yet will have a file
$PRODIR/history/tapes_moved.WMXXXX made  which contains the list of files on
that tape.  These files are used to make the injobs file.  If you want to
process a special tape, you can make your own
$PRODIR/history/tapes_moved.XXXXXX file  and it will go into injobs the next
time the production manager wakes up and looks around.   

WHAT HAPPENS WHEN A FILE HAS BEEN PROCESSED?
_____________________________________________

The file gets spooled onto tape or disk, but equally importantly,

1) The DST file -> D0FS

2) RCP files for both DST and STA -> FATMEN database

3) the files /$PRODIR/pdbkd/1994/staf.dir and 
             /$PRODIR/pdbkd/1994/dstf.dir 

are updated.

WHAT HAPPENS IF A JOB FAILS?
____________________________

the output files lists are compared to the injobs file.  If a file does not
appear in the output file list within 2 days of submission, it is added to the
list of unprocessed tapes for reconsideration.

IT'S ALL AUTOMATIC, WHAT DO PEOPLE DO?
______________________________________

People are need to deal with hardware failures and to assure that the data
actually make their way from the input file list to the output database.
The TPM also sends out error alerts that need to be investigated.

Common problems include 

1) Tape drive failures.  The program can automatically switch to a spare drive
but the bad drive needs to be replaced reasonably quickly, someone needs to
place a service call and make certain that the drive is replaced.

2) I/O server  crashes. There are procedures to restart the system but it needs
to be checked after startup.  The startup procedure is 'after_crash'.

3) Crashes due to bugs in the RECO code.  Here you may need help from the code
experts.

4) dbl3 data is not available yet for the data being processed.

5) General badness.  Processes sometimes just stop.   One needs to check them
several times a shift.  The TPM Operator's Manual will help with some of this.

Special tasks include

The previous applies to ALL stream data, special runs require that you set up
the correct projact and a special tapes_moved.XXXXXX file and check that the
output information reaches the appropriate database.  The two day wait for
resubmission may be too long for such projects.