# Presentation: 479781

## MSCF for Dummies 2002                        			HP Itanium-2, Linux and Everything (Don’t panic!)

- Bert de Jong
- MSCF Visualization and User Services Group

## Outline

- Login
- Compiling
- Running jobs
- Useful commands
- Allocations and accounting
- Refunds
- When things go wrong or you need / want more help

## Login on Opus

- Login using secure shell (ssh) with protocol 2
- Machine name is opus.emsl.pnl.gov
- If you get the following error message at login:
- Bad packet length 1349676916
- You tried to login using protocol 1
- Use ssh2 or ssh –2 to login

## File systems on Opus

- File systems available to users
- NFS file system /home/*<username> *contains users’ home directory
- /scratch (280 Gbyte) file system is local to each node, gets cleaned after job finishes
- /dtemp (18 Tbyte) file system can be seen by all
-    nodes, volatile (gets cleaned when
-    full, and on regular basis, no backups)
- Users also have access to the
- 	AFS file system, i.e. /msrc

## Compiling (1)

- Available compilers
- Compilers can be called using
*efc* (Fortran77, 90 and pieces of 95)
*ecc* (C and C++)
- Both version 6.0 and 7.0 are available. Compiling with the 7.0 produces faster code, but there might be some issues with the MLIB libraries
- Compiler flags:
*-tpp2* : optimize for Itanium-2 processor
*-O2* : default optimization level. *–O3* does not always produce faster code
*-ftz* : flushes denormal results to zero, catches some floating point assists
*-i4* : integer*4 is the default of efc. Use *–i8* to run with 64-bit integers
*-FR* : for free format Fortran (F90/F95)

## Compiling (2)

- Communication between processors via MPI
**-I/usr/lib/mpi/include** provides path to necessary include files
**-L/usr/lib/mpi/lib -lmpi -lelan3 -lelan** links in necessary libraries
- In some cases the linker had a hard time finding the mpi_init_ object : Resolve by adding /usr/lib/mpi/lib/libmpi.a to linker line
- Linker might complain about some unresolved symbols (f_arg, etc.):
- 	Add the -L/usr/lib/gcc-lib/ia64-redhat-linux/2.96 -lgcc -lg2c to linker line
- Portability library linked in using **-Vaxlib**

## Compiling (3)

- HP MLIB fast matrix and vector routines
- Contains BLAS 1-3, LAPACK, and other routines
- Link using:
- 32-bit (I.e -i4) : -L/opt/mlib/lib/linux –lveclib –llapack –lguide –lpthread
- 64-bit (I.e –i8) : -L/opt/mlib/lib/linux –lveclib8 –llapack8 –lguide –lpthread
- Library takes advantage of SMP parallelism using OpenMP
- To exploit this use –openmp when linking to MLIB
- Set MLIB_NUMBER_OF_THREADS to a number greater than 1
- Set OMP_NUM_THREADS, which is upper bound for parallelism when set greater than MLIB_NUMBER_OF_THREADS

## Running jobs

- Building job scripts, we go into some detail
- Running in batch
- Running interactively
- Some paths you might want to include in your .login or .cshrc:
- /home/scicons/scripts	: contains some tools to build job scripts
- /home/scicons/apps	: contains supported software (ADF, others)
- /home/scicons/libs 	: contains supported libraries (GA,
- 					  LAPACK, PETSC, others)

## Constructing a run script for a batch job (1)

- 							    Submission section
- 							    User specific section

- #!/bin/csh
- #BSUB –P *<your qbank account*>
- #BSUB –W *<wall clock limit*>
- #BSUB –n *<number of processors>*
- #BSUB –o *<your output file>*
- #BSUB –e *<your error file>*
- #BSUB –J *<job name>*
- #BSUB –u *<your email>*
- #BSUB -N
- #BSUB –extsched “rails=2”
- User defined and job specific section

**Constructing a run script for a batch job (1)**

## Constructing a run script for a batch job (2)

- Submission section
- QBANK account for first month: opususers
- QBANK account will be the same as on NWMPP1
- Wall clock in *<hh:mm>*
- You _WILL_ be charged for the whole node!

## Submission section (continued)

- Your job ID can be attached to the end of your output and error files by adding **.%J**, I.e. *output.%J*

**Constructing a run script for a batch job (3)**

- #!/bin/csh
- #BSUB –P *<your qbank account*>
- #BSUB –W *<wall clock limit*>
- #BSUB –n *<number of processors>*
- #BSUB –o *<your output file>*
- #BSUB –e *<your error file>*
- #BSUB –J *<job name>*
- #BSUB –u *<your email>*
- #BSUB -N
- #BSUB –extsched “rails=2”

## Submission section (continued)

- Put in your email address and –N tell LSF to send you an email when job is finished
- –extsched “rails=2” option defines that both Quadrics rails should be used for communication

**Constructing a run script for a batch job (4)**

- #!/bin/csh
- #BSUB –P *<your qbank account*>
- #BSUB –W *<wall clock limit*>
- #BSUB –n *<number of processors>*
- #BSUB –o *<your output file>*
- #BSUB –e *<your error file>*
- #BSUB –J *<job name>*
- #BSUB –u *<your email>*
- #BSUB -N
- #BSUB –extsched “rails=2”

## Constructing a run script for a batch job (5)

- User specific section
- prun –n <# of processors> <your program>
- Run your job on:
- /scratch : 280 Gbyte / node (140 Gbyte / processor)
- /dtemp : 18 TByte
- Don’t forget to copy the data you want to keep back from /scratch or /dtemp, data on these systems is volatile !!

## Constructing a run script for a batch job (6)

**Constructing a run script for a batch job (6)**

- User specific section (continued)
- $LSB_HOSTS contains a list of all processors assigned to user at runtime
- Print list to the error or output file, in case something goes wrong and a refund is needed
- Copying files to remote processors:
- 	    foreach node ($LSB_HOSTS)
- 		lsrcp *<local file> *$node:*</scratch/remote file>*
* *	    end
- Running unix commands on remote processors:
-         prun  *ls –l /scratch*

## Constructing a run script for a batch job (7)

- Tips for user specific section (continued)
- Execute a single program or task:
- prun *<# of processors>* *<your program name>*
- Execute multiple programs or tasks:
- prun *<# of processors>* *<program 1>*
- prun *<# of processors>* *<program 2>*
- prun *<# of processors>* *<task 3>*
- .....

**Constructing a run script for a batch job (7)**

## Submitting your batch job

**Submitting your batch job**

- Congratulations, your batch job is ready to be submitted!
- Submit your job by typing:
** ****	****bsub <** *<your batch job name>*
- When you ask for too much time or processors....
- [bert@ng127 ~/runs]$ bsub < very_long.bsub
- Too much time (0.48 hours) requested for job size (258 CPU)
- Time limit of 0 hours for 65 <= nCPU's <= 999999
- Request aborted by esub. Job not submitted.
* *

## Running interactive jobs

- For shorter and/or smaller test jobs
- bsub –P *<QBANK account>* -n *<# procs>* –Is  csh
- Some bsub job variables can be set on the command line or as environment variables
- Set LSF_DEFAULTPROJECT = *<QBANK account>*

## llnw for NWChem users

- Type ( /home/scicons/scripts/ ) llnw, helps you build and submit job script:

- NWChem interactive load leveller submission
- -------------------------------------------
- Input file name (~/nwchem.nw): *nwchem.nw*
- Full path for output (~/nwchem.out):
- Total number of processors you want to use (8): *16*
- Number of nodes you want to use (each node has 4 processors) (4): *4*
- Time limit (00:30:00): *1:00:00*
- Account to charge (pnnl0): *mscfcons*
- Load Leveller queue (batch):
- Are you doing a molecular dynamics run? (y/n): *n*
- NWChem executable (/home/nwchem/bin/nwchem):
- OK? (yes): *yes*

## I submitted my job, when will it run?

- The LSF scheduler is used to maximize the utilization of the machine, scheduling jobs and doing backfill
- Scheduler works using 				      first come, first serve strategy
- You can use backfill to your 			     advantage submitting a job small 			      and short enough to run directly
-     (check availability with **window**)

## Job running policies

- No....you cannot run a job on all 256 processors for a week!!
- Job running policies on Opus
- Number of processors:
- 	- Minimum of  4, maximum of 64
- Time limits:
- 	- 4-16 processors = 24 hours
- 	- 17-32 processors = 16 hours
- 	- 33-64 processors = 8 hours
- Maximum of 2 running jobs
- Maximum of 5 queued jobs
- For special permissions to go beyond user limits, ask Don Jones (dr.jones@pnl.gov)

## Opus vs MPP1 performance wise

- I.e., 8 processors on Opus can do the work of 64 processors on MPP1!!
- Hence, use less processors whenever you can
- Use the larger memory and disk to your advantage

|  | Opus | MPP1 | Factor |
| --- | --- | --- | --- |
| # of processors
Peak processor
Peak system
Memory per proc.
/scratch per proc. | 256
4 Gflops
1 Tflops
6 Gbyte
140 Gbyte | 512 
0.5 Gflops
1⁄4 Tflops
0.5 Gbyte
4 Gbyte | 1⁄2
8
4
12
35 |

## What is the status of my job?: showq

- Similar to NWMPP1!

**Useful commands (1)**

- [bert@ng127 ~]$ showq
- Jobid User     Stat SubmitTime  cpu RunLim StartTime   Jobname
- ===================================================================
- 35430 bylaska  RUN  12/06-07:24  64 03:53  12/06-11:46
- 35451 junli    RUN  12/06-10:40  32 13:48  12/06-15:41 26736
- 35478 freedman RUN  12/06-12:09   4 01:48  12/06-15:41 glass_test
- 35453 junli    RUN  12/06-10:43  32 14:41  12/06-16:34 26914
- 35432 bylaska  RUN  12/06-07:28  64 06:54  12/06-16:47
- 35487 d3j540   RUN  12/06-13:34  16 11:08  12/06-17:01
- 35446 bylaska  PEND 12/06-10:10  64 10:00
- 35454 junli    PEND 12/06-10:44  32 16:00              26993
- 35488 d3j540   PEND 12/06-13:34  32 06:00
- 35489 d3j540   PEND 12/06-13:34  64 04:00
- 35507 d3k958   PEND 12/06-15:26  32 16:00              blyp
- 35508 d3k958   PEND 12/06-15:26  32 16:00              b3lyps

## What is the status of my job?: bjobs –u all 

- bjobs alone will give you a list of your jobs !

**Useful commands (2)**

- [bert@ng127 ~]$ bjobs -u all
- JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
- 35478   freedma RUN   normal     ng126       2*ng106     glass_test Dec  6 12:09
-                                              2*ng105
- 35487   d3j540  RUN   normal     ng126       2*ng121     *How many  Dec  6 13:34
-                                              2*ng112
-                                              2*ng107
-                                              2*ng108
-                                              2*ng88
-                                              2*ng110
-                                              2*ng109
-                                              2*ng111
- 35446   bylaska PEND  normal     ng126                   *129.02.nw Dec  6 10:10
- 35454   junli   PEND  normal     ng127                   26993      Dec  6 10:44
- 35509   d3k958  PEND  normal     ng127                   uhf        Dec  6 15:26
- 35514   rory    PEND  normal     ng127                   ccm        Dec  6 17:19

## What is the status of the machine?: rinfo

**Useful commands (3)**

- [bert@ng127 ~]$ rinfo
- MACHINE     CONFIGURATION
- ng          lsf
- PARTITION       CPUS    STATUS        TIME   TIMELIMIT  NODES
- root             260                                    hng[0-1] ng[0-127]
- parallel     212/254   running 12:19:41:24              ng[0-38,40-127]
- RESOURCE        CPUS    STATUS        TIME    USERNAME  NODES
- parallel.18258    64 allocated    06:07:35     bylaska  ng[16-31,41-47,64,113-120]
- parallel.18300    32 allocated    02:12:53       junli  ng[89-104]
- parallel.18301     4 allocated    02:12:53    freedman  ng[105-106]
- parallel.18312    64 allocated    01:06:46     bylaska  ng[40,48-63,65-79]
- parallel.18317    16 allocated       52:37      d3j540  ng[88,107-112,121]
- JOB             CPUS    STATUS        TIME    USERNAME  NODES
- parallel.19100     1   running    06:07:32     bylaska  ng16
- parallel.19101    64   running    06:07:32     bylaska  ng[16-31,41-47,64,113-120]
- parallel.19178     1   running    02:11:41       junli  ng89

## Useful commands (4)

- If you want to cancel your job!: **bkill**
- Modifying your job on the fly: **bmod**
- More processors needed
- Longer / shorter run time needed

- [bert@ng127 ~]$ showq
- Jobid User     Stat SubmitTime  cpu RunLim StartTime   Jobname
- =============================================================================
- 35430 bylaska  RUN  12/06-07:24  64 03:53  12/06-11:46
- 35451 junli    RUN  12/06-10:40  32 13:48  12/06-15:41 26736
- 35478 freedman RUN  12/06-12:09   4 01:48  12/06-15:41 glass_test
- 35453 junli    RUN  12/06-10:43  32 14:41  12/06-16:34 26914
- 35432 bylaska  RUN  12/06-07:28  64 06:54  12/06-16:47
- 35487 d3j540   RUN  12/06-13:34  16 11:08  12/06-17:01

## Useful commands (5)

- How many nodes are used or available?: **window**
- Other info about the machine:
** ****/etc/motd **: contains login message

- [bert@ng127 ~]$ window
- Backfill Window for user: bert
- 8 processors (4 nodes) are available with no timelimit
- 30 processors (15 nodes) are available with timelimit 01:46:48
- 212 processors (106 nodes) are not available because they are busy
- 2 processors (1 nodes) are not available because they are down
- ----------------------------------------
- 252 processors (126 nodes) total

## Allocations and accounting

- Your allocation of time is in the bank, QBank that is 
- QBank will do the accounting & withdrawals for you
- Commands to query your allocation:
- qbalance -h		: gives remaining time in cpu-hours
- qlsuser			: gives a list of all account available 				  to you
- qtxns –u *<username>*	: gives overview of
- 					  all your withdrawals
- Remember: you will be charged for all
- 	processors dedicated to you

## Job refunds

- When are you eligible for a refund:
- Hardware failures
- System software failures
- Certain user software cases
- If you want to apply for a refund:
- Send message to MSCF Consulting queue
- Save the appropriate input, output, and error file; we need to verify the source of the problem before giving refund
- Refund based on time lost; partial refund given in cases where restarts were possible

## Available program packages and libraries (1)              

- Supported program packages available on Opus
- Location is /home/scicons/apps
- We will be making codes available over the next month
- Codes:
- ADF
- STAR-CD
- MOPAC 2000
- MOLPRO
- CCM
- MM5
- NWChem is located in /home/nwchem/bin

## Available program packages and libraries (2)

- Supported libraries available on Opus
- We will be making libraries available over the next month
- Libraries in /home/scicons/libs
- Global Array tools (GA)
- PETSC
- SCALAPACK
- New libraries and programs can be added on request, depending on general usefulness and support

## When things still go wrong?

- When things go wrong, who you gonna call?
- MSCF general web site with lots of info
- 	_*http://mscf.emsl.pnl.gov/*_
- Quick reference card (soon)
- MSCF Visualization and
- 	User Services group (VISUS)
- For refunds
- General help running jobs
- Assistance getting your code to run (in parallel)
- Scientific consulting: mscf-consulting@emsl.pnl.gov