# Presentation: 701061

## PathfinderWorkload Analysis of PDSF Batch System - Tool in Queue and Cluster Design

**Pathfinder****Workload Analysis of PDSF Batch System - Tool in Queue and Cluster Design**

**Iwona Sakrejda**
**UC Berkeley cs 266**
**05/07/2001**

## Outline

- Introduction
- PDSF Cluster Layout
- Workload Description
- Characteristics of the Batch System (LSF)
- Workload Monitoring Tools
- Analysis of Data for period 01/01 - 04/01
- Impact of Analysis on Purchasing Profile.

## Profile of PDSF Users

- High Energy and Nuclear Physics Theory and Experiments

## Profile of PDSF Users (cd)

- Important role of computing in HENP
- Reduced funding profile
- Growing size and complexity of data (1GeV ->10TeV)
- Highly parallel structure of analysis

- Loosely connected commodity hardware and software  a reasonable solution.
- Batch systems used to coordinate and integrate independent nodes into a computing system.

## PDSF Cluster

- Lawrence Berkeley National Laboratory
- National Energy Research Scientific Computing Center
- Energy Sciences Network
- Parallel Distributed Systems Facility

- 2 box cars full of hardware from SSC - only name left.

## PDSF CLUSTER

**1.2TB**

- Veritas

- Vol Mgr

- Interactive Nodes (8)

**6 GB**
**Scratch/node**

- 2 CPU/node
- 333 MHz/PII

- Batch Nodes (150)
- LSF 4.1
- ~250 CPU’s total
- 400MHz, 450MHz,
-  650MHz, 266MHz

- 14 Data Vaults (~7.5TB)

- Sun E450

- Sun
- Ultra 60
- (AFS)

**CISCO**
**5500**

**HPSS**

- 100Mb network
- some      GigE

**PDSF CLUSTER**

- High Bandwidth Nodes (10 2-processor)

## WORKLOAD

## PDSF Load Profile

- 300TB/year of raw data - raw data rates = 3 X transfer rates
- only 10% of raw data transferred  + all DST’s (data summary tapes - not tapes at all!!!)
- Monte Carlo (MC) simulations of the detector response and subsequent event reconstruction.
- negligible input
- long processing time
- 1GB output files
- Analysis of DST’s
- intense read through all files in a set (~200-300GB in one run)
- moderate to small CPU needs
- small output
- Embedding. MC data is mixed with raw data and reconstructed.
- ~1GB input
- long processing
- ~1GB output.

## - Batch System for PDSF

**- Batch System for PDSF**

- Manages load on ~200 heterogeneous nodes integrating them into a computing system.
- Expensive software to run on commodity hardware for poor users (~$30 000).
- Easy to set up and manage.
- Popular in HENP community - users are familiar with it.
- Share allocation.
- SHARE_INFO_FOR: pdsf-clust/
- USER/GROUP   SHARES  PRIORITY  STARTED  RESERVED  CPU_TIME  RUN_TIME
- e871           20       6.667      0        0         0.0        0
- amanda         15       5.000      0        0         0.0        0
- rhicthry        1       0.333      0        0         0.0        0
- sno            10       0.105      3        0    112038.0   316011
- rhstar         50       0.024     99        0   1792390.0  7378536
- Provides detailed accounting information.

## PDSF Cluster Monitoring

##  -PDSF Load Monitoring

- Monitoring of network services (SMTP, POP3, HTTP, NNTP, PING, etc.)
- Monitoring of host resources (processor load, disk usage, etc.)

- ~200 jobs accessing 1 data vault
- in R/W mode
- benefits of sequential I/O erased
- dv thrashing
- dvio resource introduced to limit
- the problem

## - Batch System Monitoring

- pdsflx004 56% lsload
- HOST_NAME       status  r15s   r1m  r15m   ut    pg  ls    it   tmp   swp   mem
- pdsflx004.nersc     ok   0.0   0.0   0.0   1%   0.2   9     0   73M 2034M  437M
- pdsflx018.nersc     ok   0.0   0.0   0.0   0%   0.0   0 19584   97M 2043M  479M
- pdsflx012.nersc     ok   0.0   0.0   0.0   1%   0.1   0 30816  100M 2043M  479M
- pdsflx074.nersc     ok   0.0   0.0   0.0   0%   0.1   0 46592   87M 2043M  479M
- pdsflx022.nersc     ok   1.0   1.0   0.9 100% 161.4   0 46656   91M 2043M  591M
- pdsflx028.nersc     ok   1.0   1.0   0.9 100% 162.5   0 46656   92M 2044M  591M
- pdsflx027.nersc     ok   1.0   1.0   0.8 100% 162.6   0 46656   94M 2043M  591M
- pdsflx155.nersc     ok   2.0   2.1   1.6 100% **701.0**   0   817   83M  255M 1446M

- r15s	The 15-second exponentially averaged CPU run queue length.
- r1m	The 1-minute exponentially averaged CPU run  queue length.
- r15m	The 15-minute exponentially averaged CPU run queue length
- ut	The CPU utilization  exponentially  averaged  over the last minute,                 between 0 and 1.
- pg	The memory paging rate exponentially averaged over the last minute, in pages/s
- ls	The number of current login users.
- it	On UNIX, the idle time of the host  (kbd  not touched on all sessions), in minutes.
- swp	The  amount of swap space available, in megabytes.
- mem	The amount of available memory, in megabytes.
- tmp	The amount of free space in /tmp, in megabytes.

## - Batch System Monitoring

- 1 min paging rate

## - Batch System Monitoring

- SHARE_INFO_FOR: pdsf-clust/
- USER/GROUP   SHARES  PRIORITY  STARTED  RESERVED  CPU_TIME     RUN_TIME
- e871           	20	6.667      	0       	   0         		0.0        		0
- amanda       	15       	5.000      0        	   0         		0.0        		0
- others          	1       	0.333      0        	   0         		3.0        		0
- atlas           	1      	0.333      0        	   0         		0.0        		0
- e895            	1       	0.333      0        	   0         		0.0        		0
- rhicthry        	1       	0.333      0        	   0         		0.0        		0
- admin           	1      	0.333      0        	   0         		0.0        		0
- sno            	10      	0.105      3        	   0    	      112038.0   	      316011
- rhstar         	50       	0.024     99        	   0   	    1792390.0  	   7378536
- dynamic priority = number_shares / (cpu_time * CPU_TIME_FACTOR + run_time * RUN_TIME_FACTOR + (1 + job_slots) * RUN_JOB_FACTOR)
- Gresham's law - bad jobs drive out good jobs in a process of adverse selection.

## - Batch System Accounting

- Each job upon completion writes a record into lsb.acct file
- "JOB_FINISH" "4.1" 989205863 617942 14819 40108562 1 989189416 0 0 989205588 "pfachini" "short" "linux select[defined(dv08io)] rusage[dv08io=50]" "" "”
- "pdsflx003.nersc.gov" "HbtFlow" "" "st_physics_1231007_raw_0036.event.root.flowpicoevent.root.out" "" "989189416.617942" 0 1 "pdsflx007.nersc.gov" 64 5.0 "" "root4star -q -b StHbtFlowLikeSignRhoExBIIM.C\(1e9,\""-\"",\""/auto/pdsfdv08/starspec/pfachini/pDST/global/P00hm/minbias/st_physics_1231007_raw_0036.event.root.flowpicoevent.root\"",\""st_physics_1231007_raw_0036.event.root.flowpicoevent.root\""\)" 245.960000 8.340000 0 0 -1 0 0 15448 8607 0 0 0 -1 0 0 0 0 0 -1 "pfachini" "default" 0 1 "/usr/local/bin/tcsh" "" 0 68072 106352 "" ""
- Many fields empty or 0, others (like max virtual memory) inaccurate (lnx).
- LSF provides Analyzer (a licensing fee per node, expensive)
- Records analyzed and loaded into MySQL Data Base.

## Batch System Logs

- mysql> show columns from lsf_acct;
- +----------------------+---------------+------+-----+---------+-------+
- | Field                | Type          | Null | Key | Default | Extra |
- +----------------------+---------------+------+-----+---------+-------+
- | modify_date          | timestamp(14) | YES  |     | NULL    |       |
- | event_type           | varchar(12)   |      |     |         |       |
- | version              | varchar(10)   |      |     |         |       |
- | event_logged         | datetime      | YES  |     | NULL    |       |
- | job_id               | int(11)       |      |     | 0       |       |
- | user_id              | int(11)       |      | MUL | 0       |       |
- | bit_flags            | int(11)       |      |     | 0       |       |
- | processors_request   | smallint(6)   |      |     | 0       |       |
- | submission_time      | datetime      | YES  |     | NULL    |       |
- pdsflx004 61% more runTime
- select event_logged, dispatch_time,  unix_timestamp(event_logged), unix_timestamp(dispatch_time), unix_timestamp(submission_time), user_time, user_group,  pages_fault, execution_host_list from lsf_acct where unix_timestamp(submission_time)>983433600 and  unix_timestamp(event_logged)<986112500 and user_time>0 order by user_group;
- Perl script analyzes output from query.

## PDSF Batch System Performance 02/06/2001 - 03/01/2001

- Number of CPU's available during the accounting period - 225.
- Number of CPU seconds available on the cluster - 1998258 (23.1 days) x 225 = 5197.5 days.
- Number of jobs run during that time - 62 035.
- Wall clock time used by the jobs - 4547.5 days (87% utilization).
- CPU time used by the jobs - 3547 days (78%efficiency).
- Project	Number of jobs	Wall clock          % Wall Clock	CPU time(s) Efficiency(%)
- admin	       172		90371s       25.1h	0.02	              8	      0.01
- amanda	       648	             775695s        215h	0.19	              1147983	       ?
- deepsrch	           6	                     26s	               0.00	              0.21	      0.07
- E871	         73	             204279s       56.7h	0.05	             198058	      96.95
- E895	       421	          23857152s   6627.0h	6.07	             23830344	      99.89
- Na49	      274	            1736794s     482.4h	0.44	             333536           19.20
- rhstar	  59070	        187420429s  52061.2h	47.70	             101781672	      54.31
- rhicthry	         10	            1376697s     382.4h	 0.35	             621558	      45.15
- sno	     1542	        177414698s  49281.9h	45.16	             175166427      98.73

## PDSF Batch System Performance 03/01/2001 - 03/31/2001 

- Number of CPU's available during the accounting period - 225.
- Number of CPU seconds available - 2678400 (31 days) x 225 = 6975 days.
- Number of jobs run during that time - 33 049.
- Wall clock time used by the jobs - 3644.9 days (52% utilization).
- CPU time used by the jobs - 1895.2 days (52%efficiency).
- Project	#of Jobs	    Wall Clock   % Wall Clock    CPU               Efficiency    Swapping
- admin	172	     126.6h	0.1	434532	            95.4	1
- amanda	1983	   1878.6h	2.0	7622334	           112.7	3
- atlas	27	       10.3h		22929	            87.6	24
- deepsrch	48	         7.3h		13433	            69.8	189
- E871	51	       71.2h		234602	            91.5	1
- E895	23	     103.0h	0.1	36109	            98.9	0.1
- Gc5	3	           9s
- HEP	4	         29s
- NA49	67	     104.1h  	0.1	147471	            39.3	90
- rhstar	26390     73538.1h	             84.2	114184320           43.1	170
- rhicthry	176	  3158.9h	               3.6	11165167	            98.2	2
- sno	4016	 8484.3h	               9.6	29549993           96.7	13

## Conclusions

- Workload analysis of LSF monitoring and logs proved to be a useful tool in optimizing cluster performance and deciding about purchasing.
- dvio resource established to moderate disk vault access
- scratch resource provided to facilitate data copying to local disks  for up to 10GB
- more memory ordered for the batch nodes to minimize swapping
- high bandwidth nodes recommended for FY2002 (300GB scratch, 2GB memory)