From:	GENSER       "Krzysztof L. Genser; (708)840-8278; MS 357"  7-OCT-1994 00:16:34.90
To:	SCHELLMAN,DOROTA,LUEKING
CC:	GENSER
Subj:	monitoring "d0$data$buffer", draft

                                                         Fermilab,  6-OCT-1994

Monitoring Files on D0$DATA$BUFFER

(it was originally  written for data aids not  associated with the farm,
so farmers please excuse some specific language)

Introduction

The D0$DATA$BUFFER (buffer) area  is used to transfer (mostly DST) files
from  the  "outside  world" to  D0FS.  The  files come  mainly  from the
production farm.

The buffer is split into three sub areas:

  "D0$DATA$BUFFER01" = "D0FS_D0TSAR$DATA1:[DATA01]"
  "D0$DATA$BUFFER02" = "D0FS_D0TSAR$DATA1:[DATA02]"
  "D0$DATA$BUFFER03" = "D0FS_D0TSAR$DATA1:[DATA03]"

One for each farm I/O node.

There are four FMD0 jobs:

  Entry  Jobname         Username     Blocks  Status
  -----  -------         --------     ------  ------
    325  FM_DST_COPY_SERVER01
                         FMD0                 Executing
         On available batch queue D0TSAR_FATMEN_BATCH

    326  FM_DST_COPY_SERVER02
                         FMD0                 Executing
         On available batch queue D0TSAR_FATMEN_BATCH

    328  FM_DST_COPY_SERVER03
                         FMD0                 Executing

    331  FM_DST_COPY_SERVER05
                         FMD0                 Executing
         On available batch queue D0TSAR_FATMEN_BATCH

which move files from buffer to  other D0FS areas, mainly to D0_DATA_DST
but also e.g. to D0_DATA_MC.  FM_DST_COPY_SERVER05 copies files from the
PROMANXX:[FM_BUFFER] area.

The jobs  run most of  the time  with some  rest arround  4-5am when the
backup of the FATMEN area is done.

Moving files is steered by the RCP  files which accompany them. The rule
of the thumb is that if the file  name is A.B the RCP file will be named
A.BRCP. It is also possible the RCP file will be like PREFIXA.BRCP

FM_DST_COPY_SERVERxxs  do  not "see"  files other than  the RCP ones. It
takes the RCP  file, reads  it in and then  tries to  move the data file
reffered to in the RCP file.

If FM_DST_COPY_SERVERxxs have some  problems with either the RCP file or
the file the RCP file reffers to,  it marks the files (ususally RCP ones
only) with a WG_X_ prefix where X has the following meaning:

 WG_T - bad target
 WG_N - no dst itself
 WG_E - dst exists on the target area
 WG_D - rcp archive area problem
 WG_C - bad copy to the target area
 WG_F - bad fm check
 WG_P - bad pdb check
 WG_B - bad pdb copy
 WG_R - bad fm entry
 WG_M - should not happen

Monitoring

0.) make sure all the four servers are runing, do
 $ show entry/user=fmd0 *copy*

1.) Make sure the number of free blocks on buffer is greater than 4M
 and does not decrease steadily over few days;
 $ show dev d0$data$buffer

2.) Observe the appearance of the recent WG_%_ files:
 $ dir D0FS_D0TSAR$DATA1:[DATA0%]WG_%_*/since=-1-0/mod/size=all
 try to find out why were they classified as wrong by by analysing 
 the log files:
 $ edit/tpu/read FATMEN_ROOT:[FMD0.LOG]FM_DST_COPY_SERVER%%.LOG
 you may need to analyse also previous versions of the log file.
 (Always read log files in the read only mode!).
 
3.) Make sure the number of correct non RCP files older than 3 days is
 small and that they do not occupy to much space:
 $ dir D0FS_D0TSAR$DATA1:[DATA0%]/before=-3-0/mod/size=all/exc=(*.*RCP,wg_%_*.*)
 If the number is not small try to find out why it is so by analysing
 the log files:
 $ edit/tpu/read FATMEN_ROOT:[FMD0.LOG]FM_DST_COPY_SERVER%%.LOG
 or by tring to see if there are corresponding RCP files, or if the
 files are not on D0_DATA_DST, D0_DATA_MC, etc... If the files
 are real orphans send info to people who created the files;

4.) Make sure the number of WG_%_ files older than 3 days is small:
 $ dir D0FS_D0TSAR$DATA1:[DATA0%]WG_%_*/before=-3-0/mod/size=all

Let me know if it needs to be more clear etc...,

       Krzysztof