Command to run D0 framework executable.

Usage: rund0exe [...]

Command line options:

    General
    -------

    -h       - Get help.


    Global parameters
    -----------------

    -exe=exe           - Name of executable
    -rcp=rcp           - Name of framework rcp script
    -rcppkg=rcppkg     - Name of framework rcp package

    -outpath           - Send the output to this directory. The default is the current directory

    -initscript=script        - Initialization script for executable (executed script)
    -userscript=script        - User provided initialization script for executable (executed script)
    -sourcescript=script      - Initialization script for executable (sourced script)
    -usersourcescript=script  - User provided initialization script for executable (sourced script)
    -userpostscript=script    - User provided finalization script 
		                after executable - must be in $PWD

    -localbuild        - Use local build instead of official version
    -localrcp          - Use local rcp's in addition to official ones
    -localfwkrcp       - Use local version of main framework rcp
    -localrundata      - Use $SRT_PRIVATE_CONTEXT/rundata properly whne not using the -localrcp option
    -local             - Same as -localbuild -localrcp -localfwkrcp -localrcp
    -name=name         - Override default job name
    -rcpfiledb[=<ver>] - Set up a particular version of rcpfiledb. With no argument use the current
                       - version. Otherwise use the specified version.

    Executables built with shared library support
    ---------------------------------------------

    -shlib             - Use the version of the executable built using shared libraries
                         To build an executable using shared libraries you should do
                         the following:
                         setup D0RunII ........ [ -O SRT_QUAL=maxopt]
                         setenv LINK_SHARED yes
                         gmake all
                         rund0exe -shlib .....................


    Run Time Environment options
    ----------------------------

    -userte            - Use run time environment (rte) directory
    -rtepkg=<rteexe>   - Name (lc) of exe (yes!) within rte to be invoked
			 (rtepkg will be changed to rteexe in a future release)
    -rtepath=<path>    - Path from current working directory to rte directory.
			 if submitting to remote node this will be tarred and
			 copied to the remote node unless -rtecached is set 
    -rtecached	       - use if submitting to remote node which has a local
			 copy. If so, -rtepath should be the local path 
                         on remote node.
		
	(if you have not setup D0RunII, you need to set some variables:

	 setenv SRT_PUBLIC_CONTEXT p14.02.00  // or whatever is current
	 setenv SRT_SUBDIR Linux2.4-KCC_4_0 
	 setenv MACH_OS Linux

	as d0tools likes to know what it is doing...
	)

    Modes of running  (default mode is interactive)
    ----------------

    -batch           - Use batch queue [default = short]
    -debug=debugger  - Use interactive debugger: available options
                       totalview (recommended for p14), ddd (recommended
                       for p16 and later), gdb.
    -maxopt          - Use optimized build.
    -maxopt-strip    - Use optimized stripped build (smaller)
    -jobname=jobname - Specify batch job name [default = rund0exe]


    clued0 (PBS) batch system switches:
    ------

    -cput=time - Specify maximum CPU time. [default = 3:00:00]
    -mem=mem   - Specify maximum memory. [default = 500Mb]
    -node=node - Specify node where job should be run [default = system chooses]
    -thisnode  - Run job on the current node.
    -nomail    - Tell PBS not to send E-mails to the user upon job completion.


    cab (PBS) switches:
    ---

    -cabsrv1   - Use the CAB (central analysis backend) batch system. 
    -cabsrv2     Detailed documentation available at:
                 http://www.nuhep.nwu.edu/~schellma/cab/cab_doc_v2.html
                 CAB can get input either from SAM (preferred),
                 cross mounted disks (project disks or home directories),
                 or local disks. In the last case files are transferred
                 to CAB prior to the job start (with a 5 GB limit).
    -fzk       - Use CAB style jobs at the computing center in Karlshruhe (FZK)
                 You must be logged on d0.fzk.de to submit jobs to FZK

    -q=<queue> - submit to a specific batch queue

    -scratch=<location>  - need to have this as you have to make a
			   tar file with your code in it.
		           Output is returned to this area as well

    -cabtar  - use a pre-existing tarball which is already in -scratch
	       DO NOT INCLUDE THE PATH! If cabtar is not set, a tarball
	       is made from the contents of $PWD. To prevent the 
               creation of the tarball use -cabtar=empty (but then
               you are limited to the official compiled version of
               the D0 code and the official RCP parameters).

    -jobs=num - number of parallel jobs you can run with single consumer
	        works only with SAM input.

    -caboutpath - path for the output, defaults to -scratch (synonym for -outpath)

    -cabouthost - host for the output path, defaults to submission node
 
    -cabmonitor - check every 5 minutes the events.read file
                  and print the total number of events read so
                  far and the name of the last opened file in
                  the job logfile

    -cabusenfs - when combined with the -filelist option read the
                 input files through NFS without making a local 
                 copy (works only with input from project
                 disks and from home directories)

    -cabskipchecks - do not perform checks on input files when
                 reading from disk (ONLY FOR ADVANCED USERS, 
                 to be used with -filelist -cabusenfs)

    -cabproduction - allow the use to run separately the SAM start
                 project, the SAM stop project and the workers (use
                 start, stop or continue as options). When the start
                 or stop options are used no worker is submitted. When
                 you use this option you have to ensure that the workers
                 are started after the SAM project has been properly
                 started. And stop the project only when you're sure
                 that all the workers have finished.

    -cabnodetype - specify properties for the nodes the batch jobs are 
                   to run on. Multiple values should be separated by ':'.
                   The special value 'any' will override any defaults and 
                   force an empty property list.

    -cabdebug - debug the job submission to CAB, extensive printouts
                no jobs are submitted

    -nomail    - Tell PBS not to send E-mails to the user upon job completion.

    Special *temporary* CAB feature: If you only want to include the ONE
    executable specified via your run command in your CAB tar file
    (thereby reducing its size and speeding up transfer), set the
    environmental variable CABTARSINGLEEXE to a non-null value.

    Options controlling input / output
    ----------------------------------

    -filelist=filelist.dat - List of files/datasets to process

    -defname=defname - SAM project definition name
    -filecut=num - Maximum number of files to be delivered in a
                   SAM project
    -jobfilelimit=num - Maximum numer of files to deliver to a single job. This 
                        does not work with older releases.
    -out=file  - Write processed events to specified file


    Options to control which events are processed
    ---------------------------------------------

    -num=num   - Specify number of events to process [default = all]
    -skip=num  - Specify number of events to skip [default = none]
    -eventlist=eventlist.dat  - Process only these specific collision ids
                                eventlist.dat is a file containing a list
                                of <run> <event> collision ids.
    -skipruns=badruns.dat - Skip these runs
                            badruns.dat is a file containing a list of run numbers
    -skipevents=badevents.dat - Skip these specific collision ids
                                badevents.dat is a file containing a list
                                of <run> <event> collision ids.

    Other framework options
    -----------------------

    -suppress  - Disable printing of individual error message in log file
                 (Instead, a summary is printed at the end of the job)
    -memreport - Enable framework memory leak checking (turns on framework
                 switches -memusage -memreport)
    -filter_triggers=trg - Specify a list of L3 trigger names (separated
                 by commas) used to filter events during the processing
                 (requires that the executable is linked with RegTriggerFilter)
    -copyexe   - Copy the execuable to the execution directory (instead of 
                 symlinking it). This allows rebuilding of the executable while
                 a job is running (this option is ignored when running on CAB).


    SAM options
    -----------

    -group=group - Specify SAM group [default = dzero]
    -datasetversion=version - Specify version of the SAM dataset [default = new]


    Special options
    ---------------

    -idletimeout=<hours> - Kill the job after it has been idle for 
                           the specified number of hours. A time of 0 
                           disables the monitoring. (An idle job is one 
                           which has not updated its out or log files.)

    -fpe   - Turn on floating point exception handling on Linux
    -nofpe - Disable floating point exception handling on Linux

    -runscript=script  -  Specify a custom script to run 

        The specified script will be run before actually running the
        executable.  Note that this script will only work reliably if it
        contains command line commands.  It is intended for advanced users.

    -fwkparams - Pass any set of parameters to a framework executable

        You may pass any set of parameters to the framework executable you
        are running by using the -fwkparams option.  This option MUST be
        the last one on the command line, followed by the actual
        parameters you want to pass to the framework.  For example,

          rund0exe ... -fwkparams -option1 xxx -option2 yyy


    Expert options
    --------------

    -v       - Print out what would be done, but don't do it
    -t0      - minimal talkiness (default) (debug echo)
    -t1      - a little more talky
    -t2      - even more talky


    Deprecated options
    ------------------

    -forcercpoutfile  - Use this switch to force the usage of the output files
                        specified in the WriteEvent rcp (ignore -out switch)

       With v02-04-22, the default is to have this behavior.  It is
       overridden if the user uses the "-out" option.

Description
-----------

`rund0exe' can run any D0 framework executable.  The user must specify
the executable (-exe), the main framework rcp file (-rcp) and the
package where that rcp file resides (-rcppkg).

It is assumed that the D0 environment has been setup correctly by the
user (e.g. setup D0RunII xx.xx.xx) before running or that the user has
supplied a run time environment and specified that via -userte, -rtepkg,
and -rtepath.

The command creates a new directory where the executable will be run,
and all output from the job will be placed.  By default, the name of
the directory is a combination of executable name, release version and
input file name.  To override this default name, use the -name option.

The command supports running the executable interactively (default) or
in a batch queue (-batch). In addition, it supports running debug mode
(-debug).

To run in batch mode, use the -batch option (only available on
platforms with a batch system).  The default batch queue is 'short'.
To override this default, use the -q=xxx option.

To use the debug mode, use the -debug option.  You must have first
setup the totalview or the ddd debuggers. Use -debug=totalview to
debug with totalview, -debug=ddd to debug with DDD and -debug=gdb
to debug with GDB.

The command supports processing files either available on local disk,
or available within SAM.  To process a set of local disk files, create
an ASCII file (e.g. mydata.dat) containing a list of files to process
(one line per file).  Then specify this list using the -filelist
option (e.g. -filelist=mydata.dat).  NOTE: you must specify the
complete path of each file.  For example, if you wish to process the
files myfile1.dat and myfile2.dat, which are located in
/scratch/1/mydir, you would include the following two lines in your
"mydata.dat" file:

	/scratch/1/mydir/myfile1.dat
	/scratch/1/mydir/myfile2.dat


To process files within SAM, first use SAM to create a dataset
definition, and then use the -defname option.  For convenience, users
may optionally place the SAM dataset definition name into an ASCII
file, prefacing it with the string "SAMDEFNAME:", and then use the
-filelist option.

To specify the number of events to process, use the -num option
(e.g. -num=100).  To skip a certain number of events, use the -skip
option.

If you would like to process only specific collision ids (<run>
<event>), you may use the -eventlist option.  Create a file with a
list of desired collision ids and use -eventlist=thatfile.dat.  For
example, to process event number 123456 in run 191919, create a file
(eg. myevents.dat) with the line "191919 123456" and use
-eventlist=myevents.dat.

By default, the specified framework executable must reside in the
official release. To use a locally built exe, use the -localbuild.
When using this option, you should execute this command at the base of
that local build.

If you would like to use the executable in the official build, but
make a few changes to some rcp parameters in your local area, use the
-localrcp option.  For example, if you would like to change a
parameter in a rcp file from package xxx, first checkout that
package's rcp directory using the command "cvs checkout xxx/rcp" and
modify your local copy. Then issue the rund0exe command with the
-localrcp option.

When -userte is specified, the framework rcp's for the executable (!) 
specified by -rtepkg are required to be within the runtime environment
directory at runtime/framework_rcps/<executable>.  E.g., for D0reco_x,
you would have -rtepkg=d0reco_x and the rcp's would be at
runtime/framework_rcps/d0reco_x
Example - reco_analyze
----------------------

To run reco_analyze out of the p05.00.04 production release,

  % setup D0RunII p05.00.04

  % rund0exe -exe=RecoAnalyze_x \
             -rcppkg=reco_analyze \
             -rcp=runRecoAnalyze.rcp \
             -filelist=mydata.dat 

where mydata.dat contains a list of files to process.  
  
To process files in SAM,

  % rund0exe -exe=RecoAnalyzeSAM_x \
             -rcppkg=reco_analyze \
             -rcp=runRecoAnalyzeSAM.rcp \
             -defname=my-dataset-definition-name 

To use a local version of an emreco rcp file,

  % cd myrundir
  % cvs checkout emreco/rcp
  
  ---  modify appropriate rcp script  --

  % rund0exe -localrcp \
             -exe=RecoAnalyze_x \
             -rcppkg=reco_analyze \
             -rcp=runRecoAnalyze.rcp \
             -filelist=mydata.dat 

NOTE: If you want to run an executable out of a production release,
and wish to modify one of the rcp parameters associated with a given
package in that release, you will need to checkout that particular
version of the package.  For example, to use p08.09.00 RECO, and
modify a parameter in emreco, do

 % setup D0RunII p08.09.00
 % cvs checkout -r p08-br emreco/rcp   # Get emreco from the "p08 branch"
  
  ---  modify appropriate rcp script  --

  % rund0exe -localrcp \
             -exe=RecoAnalyze_x \
             -rcppkg=reco_analyze \
             -rcp=runRecoAnalyze.rcp \
             -filelist=mydata.dat 

Example - rund0exe with an rte
-----------------------------

A run time environment has only the executable, such as 
D0reco_x, and the files needed by it at run time--no source code.

To run D0reco_x out of a particular run time environment
directory (named, say, rte0229 residing on /scratch/10/joeuser):

   % setup d0tools
   % rund0exe -exe=D0reco_x \
              -userte \
              -rtepkg=d0reco_x \
              -rtepath=/scratch/10/joeuser/rte0229 \
              -filelist=mydata.dat \ 
              -format=data

where the file mydata.dat in the current working directory contains 
a list of files to process. (For example, it could be the single
line (with a trailing newline, of course):

   /home/joeuser/all_0000158994_146.raw 

You would make up a run time environment directory for D0reco_x onto the
directory /scratch/10/joeuser/rte0229 as follows:

   Make a local release:

      % setup D0RunII t02.29.00
      % newrel -t t02.29.00 MyLocalRel
      % cd MyLocalRel
      % d0setwa
      % setup d0rte
      % d0rte t02.29.00 /scratch/10/joeuser/rte0229 "" rcplink d0reco_x

   At this time, you would need to use the latest version of d0reco as 
   the one built into t02.29.00 does not have everything needed for rte.
   To do this, insert the following lines after the d0setwa:

      % addpkg -h d0reco
      % gmake

   This deficiency should be removed shortly which will make this unnecessary.

   You could make a run time environment in this way for any package's 
   executable as long as it and the packages which supply it run time data
   follow the d0rte publish and subscribe rules describe further in the 
   d0rte package's documentation.

Known Bugs
----------

In order to override a framework rcp, you must checkout the package
and make a local build of the executable, and then use the -localbuild
switch.


Please report problems to melanson@fnal.gov