SDSS use of FNALU cluster

GETTING AN ACCOUNT
------- -- -------

If you do not have any accounts on any FNAL machines, you first
have to read and sign the "Fermilab Policy on Computing" located
at http://www.fnal.gov/cd/forms/policy_form.html.

If you already have an account on any FNAL machine, then we already
have your signature.  You can then sign up for an account on the
FNALU cluster at http://www.fnal.gov/cd/forms/acctreq_form.html

Please fill in the information requested.  Some hints:

The Experiment is SDSS; the Affiliation is your home institution.
For the supervisor/spokesperson, fill in Chris Stoughton with
email stoughto@fnal.gov.  (Unless I am on vacation or away,
then use Steve Kent or someone else who can approve the request.)

You are asking for an account on "UNIX Systems" so click that
box and choose fnalu from the menu directly below it.

You do not need a "project request form."  Fill in a sentence
about sdss int eh "briefly describe computing to be done"
box.

In the "Please include any additional informatin in the area below"
box, it is also helpful to request being put in the unix group SDSS.

WHAT MACHINES ARE THERE
---- -------  --- -----

Here are the Dec and SGI nodes available.
				speed	memory	swap	scratch
name	os	mode	#proc	Mhz	MB	MB	GB	DLT drives
fsgi02	IRIX+6	b+i	16	250	 512	2509	7.9	6(3)
fsgi03	IRIX+6	b+i	16	195	2816	4096	8.9	3(2)
fsgb01  IRIX+6  b        2      180      256     256     -       -
fsgb02  IRIX+6  b        2      180      256     256     -       -
fsgb03  IRIX+6  b        4      180      512    2048     -       -
                
fdei01  OSF1    b+i      4      275      621     767     -       4(3)
fdei02  OSF1    i        3      200      495     ?       10(5x2) -
fdeb01  OSF1    b        1      233       54     256     -       -
fdeb02  OSF1    b        1      233       54     256     -       -
fdeb03  OSF1    b        1      233       54     256     -       -
fdeb04  OSF1    b        1      233       54     256     -       -
fdeb05  OSF1    b        1      233       54     256     -       -
fdeb06  OSF1    b        1      233       54     256     -       -
fdeb07  OSF1    b        1      233       54     256     -       -


mode:  "b" means batch jobs only; "i" means interactive only; "b+i" means both.
scratch:  locally available scratch space under /usr/scratch/

The node fsgi03 is special.  In addition to sdss products it has idl
installed.  It also has 500 Gb of disk dedicated to eag sdss science
analysis.  Only FNAL people can write to these disks.  Under
/usr/sdss/data01/imaging there is the usual directory tree of imaging
outputs.  Everyone can read and copy these files to their home
institution or to scratch areas for processing.

If the CPU becomes saturated on fsgi03 we will ask non-FNAL people
to please use fsgi02 or fdei02 for running jobs.

PRODUCTS
--------

As with all FNAL machines, before you use a program you (usually)
have to set it up.  So if you want to use emacs, first type "setup emacs"
to get the correct version of emacs added to your path.

SDSS products are located in a different place, so you need to add
some magic to the top of your .cshrc file to let "setup" know where
they are.  Copy ~stoughto/.cshrc to your place or make your .cshrc
look like this at the top:

setenv UPS_EXTRA_DIR /afs/fnal.gov/files/code/sdss/p/upsdb
if ( -f "/afs/fnal.gov/ups/etc/setups.csh" ) then
    source "/afs/fnal.gov/ups/etc/setups.csh"
endif

The order is important.

WHY CAN'T I WRITE IN MY HOME AREA?
--- ----- - ----- -- -- ---- -----

The home areas are AFS disks, and use a security scheme which involves
tickets.  If you mysteriously can no longer write in your area, the
command "klog" will ask for your password and then give you a fresh
ticket.  

You may have to do this after logging in, and after leaving a session
idle for a long time.

WHY SHOULD I USE BATCH QUEUES?
--- ------ - --- ----- -------

This machine is shared with others at FNAL so there are a few ground
rules we have to follow.  The intention is to give you significant
compute resources for science analysis, but not the entire machine!

The first rule is that no interactive job can consume more than 30
minutes of CPU time.  There is a system job which "bags" jobs which
are longer than that.  So if you process all of a sudden goes away, it
probably got bagged.

You run longer jobs with the fbatch facility.

% setup fbatch
% fbatch_queues  -- this gives you a list of the queues.  For example,
	the queue named "4hr" has a 4 hour CPU limit.  (This might be
	normalized in some way to take into account the CPU speed.)
% fbatch_sub -q 4hr -m fsgi03 job.csh
	Submit the executable file job.csh to the 4 hr queue.  You get
	mail when it is finished.
% fbatch_jobs -- see what jobs you have running
% fbatch_peek -- take a look at the log file at it develops.


You need some magic at the beginning of your job.csh file to make
the various products work:

# To set up any product (including IDL)
setenv UPS_EXTRA_DIR /afs/fnal.gov/files/code/sdss/p/upsdb
source "/afs/fnal.gov/ups/etc/setups.csh"

# now setup what you need...
setup astrotools
setup idl


The file ~stoughto/fbatch/example.csh is a "hello world" example
that can be submitted with the command:

<fgsi02.fnal.gov> fbatch_sub -q 4hr -m fsgi02 example.csh

This will send you mail when it finishes.  See the man page
for fbatch_sub for more options.

DO I HAVE TO TYPE IN A PASSWORD FOR EACH BATCH JOB I SUBMIT?
-- - ---- -- ---- -- - -------- --- ---- ----- --- - ------

No.  If you do the following:

source $FBATCH_DIR/bin/fbatch_setpgp.csh

it will ask for your password and set an environment variable with
an encrypted ticket.

WHERE CAN I RUN IDL?
----- --- - --- ----

IDL is licensed only on fsgi03.  Various useful IDL libraries are
included in the local/ directory included in IDL_PATH when you setup
idl.

WHERE IS THE DATA?
----- -- --- -----

On fsgi03:/usr/sdss/data01/imaging there is the usual directory
structure.