FATMEN at D0

Short User's Guide

by Krzysztof L. Genser (Send remarks to FNALD0::FATMEN_MGR)

(Updated 08-NOV-1994)


*) What is FATMEN ?

FATMEN stands for Distributed File and Tape Management System

It is a file database plus database query tools and file access tools.

It uses the  concept of the so  called generic names  which point to the
real files.


*) How to use FATMEN ?

To access files simply  specify the generic  name (in upper case) as the
input file name  for CALOR_OFF,  D0USER or EXAMINE.  The rest is done by
the system for you.  Some of the above frames  accept lists of files and
in these cases one can give the  lists in the generic form. Refer to the
frame manuals for the specific details.

In order to use FATMEN make sure the program is linked correctly. Before
you link it do put  FATMEN library in  official/test/beta mode depending
on what level you want. The  specific comands are: (official mode is the
default one)

$ LIBTEST FATMEN 
$ LIBBETA FATMEN

If you do not have symbol LIBBETA defined do:
$ @D0$BETA_UTIL:SETUP_LOCAL_CMS 

If you want to avoid problems (or  simply to be able to do anything) use
FATMEN version:

FATMEN   1.85/07  940110 11:51 D0 CERN PROGRAM LIBRARY FATMEN=

or newer. If you do  not see "D0" in the  FATMEN header printed it means
you are not using the correct version. 


*) How to obtain a list of generic file names ?

Use either  FATMEN FM  shell or  FAT_LIS a  COMPACK based  query program
searching the FATMEN Catalogue and prompting you what is available.

**) How to use FAT_LIS ?

Do

$ @D0$FATMEN:SETUP
$ FAT_LIS

position the cursor using the arrow keys

press DO (PF1) key while the cursor points to "FATMEN selection"
press DO (PF1) key while the cursor points to "setting the path"
this will prompt you to select the "data origin"
position the cursor on the data origin you are interested in

TBDA  - stands for Test Beam Data
CMDA  - Data from D0 Commissioning
CLDA  - Collider Data
CLMC  - Collider MC

now you may press DO  key twice or RETURN key  once. You may want to use
HELP (PF2) key  also. The program  will present you  the current default
generic name for  the specific data  origin. Position  the cursor on the
field you would like to change. Press DO key once. 

If the program says  there are no files like  this, it means the path on
the left  side of the  cursor  excludes all the  files. You  may want to
modify the fields on the left side then. 

If the  program returns to  the same  filed changing it  or not and says
only one  element   present, this  is  exactly what  it means.  The only
element was placed in the filed you tried to change. It is OK.

If the  program  presents you  window with a  list of  names you may use
arrow keys or  PAGE_DOWN, PAGE_UP  keys to move  around it. To guide you
better the last element in the list  is shown in the message window. You
may now chose  one element  from the list  or try to  select first field
which is WILDCARD. This will prompt you for your selection.

You may use wildcards like * -  unspecified number of characters,% - one
character, but  also you may  specify ranges of runs  or partitions like
R0413(1:20)7P(1:21) etc...

Once you have established the search patch you may press BACK (PF4) key.
Then perform 
"searching the catalog"
this will prompt you for "output_file_name"
if you just press RETURN it will be FAT_LIS.GEN
After the search is successfully performed you may EXIT

You    may  now   see   which    files   were   found  by     looking in
output_file_name.GEN The file can  be defined as the input file list for
CALOR_OFF and D0USER.

FAT_LIS can be used in batch. The command sequence:
$ @D0$FATMEN:SETUP
$ FAT_LIS
$ FAT_LIS_BATCH
produces some instructions to be followed.

FAT_LIS has a limit of 10000 files to be considered for the lists. Watch
the message screen or batch log file if you did not reached the limit.


*) How to use FATMEN Shell FM ?

One can also use the FATMEN shell  FM to make more sophisticated queries
of the catalogue. It has quite  extensive help. The commands of interest
are ld, ls, cd, pwd, find, fc,  search. Doing help command will give you
usually sufficient information. Here are some command examples:

$ FM

invokes the shell

FM> LD lists directories

FM> CD //FNAL/D0/CLDA/92P1E18/DST01REU1117/ALLALL00WZEXX20

sets the default directory

FM> FC

counts files in the directory

FM> LS R043238P(01:05) -b

prints list of files with location and usage information

FM> LS R043238P(01:05) -r

checks whether files  are on the disk or not  (use with caution, opening
files via DFS takes some time...)

FM> set/mediat 1
 
sets media type to disk for the LS and SEARCH commands
 
FM> set/mediat 4,5
 
sets media type to single and double density 8mm tapes
 
FM> set/mediat 1,4,5
 
sets media type back to default
 
General hint  using FM: if  you ever hit  ^C or ^Y while  in FM exit and
enter FM  again.  KUIP (also  used by  interactive  PAW)  gets extremely
confused by ^C or ^Y.
 

*) How to copy a file to your local area ?

it is done using  the FM Find  command. This will  stage the file on the
file server if necessary. the exact syntax is:

FM> FIND generic_file_name unit_number

e.g.

FM> FIND R043229P01 88

This should create a logical e.g. FOR088 (if unit_number=88) assigned to
the file of interest. After you exit the FM 

FM> exit

you can do:

$ copy FOR088 your_newly_created_file_in_your_area

If you are planing to use many  files from a tape it is advisable to use
prestage see below.


*) How to read many tape files

**) Prestaging or the so called Server Staging Mode

The Server  Staging  Mode was  designed to  decrease the  number of tape
mounts if  more than one  file is to be  read from the  same tape and to
save the space on  the staging disk  to use it only  for the files which
are  really  needed. The  server  tries to  combine  all the  file stage
requests from the same  tape during the time  the staging job is pending
or executing, reducing  the number of mounts  and by doing so shortening
the overall staging time and diminishing the wear of tapes.
 
The Prestaging is  another concept which  allows to prepare the files to
be analysed ahead of  time, so that the  analysing job does not wait for
files to be staged or at least lets  the files to be prestaged while the
analysis job is reading the first files.
 
Now, how to invoke Prestaging. All  what needs to be done is to create a
file e.g. FAT_LIS.GEN  with the list of files  to be analysed as usually
for CALOR_OFF or D0USER and then do:
 
$ LIBTEST FATMEN                             ! if not done before
$ LIBBETA FATMEN                             ! if not done before
$ PRESTAGE FAT_LIS.GEN queue_characteristics
 
and then submit your normal job with the same queue_characteristics.

Note that  PRESTAGE just makes a  request for files  to be placed on the
staging disk. IT DOES NOT PHYSICALLY STAGE THEM.

One should keep  in mind that the  lifetime of the  files on the staging
disk is about 12h, so it does not make sense to prestage more files than
can be read within this  period of time as it  will effectively decrease
the lifetime of files even more.

*) Answers to some commonly asked questions:

**) How to do FATMEN search for files from a given run

Here is a way to find ALL stream STA files from a given run e.g. 56639
 
$ FM
 
FM> ls //FNAL/D0/CLDA/92P1E18/STA01REU%%%%/ALLALL00NONEX00/*/R056639P%% -wi
 
For  completenes this is  how to find an  INSPILL stream  RAW files from
this run:
 
FM> ls //FNAL/D0/CLDA/92P1E18/RAW01DAQ0001/INSALL00NONEX00/*/R056639P%% -wi
 
If you do not know antything more then the run number (and you do not
want to go to the controll room to find the type of run) you can try the
time consuming command:

FM> ls //FNAL/D0/CLDA/92P1E18/RAW01DAQ0001/*/*/R056639P01 -w

The same wildcarding applies when using FAT_LIS

**) How to get a list of the most recent files from a specific stream

To find  e.g.  EXP DST  files   reconstructed with  version  101% D0RECO
catalogued between Feb 1st, 1993 and Dec 31, 1995

FM> logl -1
FM> search //FNAL/D0/CLDA/92P1E18/DST01REU101%/EXPALL00NONEX00/*/* _
FM>_ cataloged=930201-951231 -gix output=FAT_LIS.GEN
FM> exit

Note the underscore "_" to indicate that the next line is a continuation
line.

The above search is very resource and time consuming.
 
The output  created is quite  similar to the  one of FAT_LIS, but
contains empty lines which can be removed using an editor.

*) Where to find (much more detailed, but not D0 specific) information ?

FATMEN is  fully  described in the  manual  (PU0131)  available from the
Computing  Division  Library  (High  Rise, 8-th  floor).   The manual is
fairly descriptive, contains index and tables of all FM commands.