# Presentation: 671266

## SSS Validation and Testing

**SSS Validation and Testing**

**September 11, 2003**
- Rockville, MD
**William McLendon**
**Neil Pundit**
**Erik DeBenedictis**

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

## 

APItest

Release Testing Experiences at Sandia

Status daemon

**APItest**
**Release Testing Experiences at Sandia**
**Status daemon**

**Overview**

## Distributed Runtime System Testing

**Complex system of interactions**
**Approach to testing**
**Component Testing**
**Benchmarks**
**Performance / Functionality**
**Operational Profile**
**Stress Testing**
**Users expect a high-degree of quality in today’s high end systems!  **

**Notes:**
A Distributed Runtime System is the ambassador between a modern parallel system and the users.  

Today, users expect a high degree of quality in the system!  This is even more important on machines at research institutions such as national laboratories or universities where queue-wait times may be large.  System managers will get calls from angry users if jobs fail to execute after being in the queue for days.

Quality is often measured in uptime... a machine that is not running costs a lot of money.  Even an open source runtime system is not free if the system is crashing due to RT system failures.  IT managers (who’s jobs are on the line) will gravitate to take the path of least job volatility.

So, how do we achieve this?  We must test extensively... but also in a smart way.  Much effort needs to be placed into unit testing.  Component reliability does have an impact on system reliability... therefore it is a step that cannot be ignored.  That is the first step to an overall structured approach to testing.

Beyond component tests, we must test applications in various ways... functionally, performance wise, via operational profiles, etc.  

## APItest

## APITEST - Overview

**Unit-testing tool for network components**
**Targeted for networked applications**
**Extensible framework**
**Dependency calculus for inter-test relationships**
**Scriptable Tests (XML Schema Grammar)**
**Multi-Protocol Support**
**TCP/IP, SSSLib, Portals, HTTP**

**Notes:**
This tool came out of discussions between myself and Narayan Desai and was presented

## Accomplishments Since Last Meeting

**Spent a week at Argonne (July)**
**Major rework of framework of APItest**
**Individual tests are atomic.**
**Framework handles the hard work of checking tests, dependencies, and aggregate results.**
**Extensibility **
**New test types are easy to create**
**Dependency System**
**Define relationships as a DAG encoded in XML.**
**Boolean dependencies on edges.**

## Supported Test Types

**sssTest**
**use ssslib to communicate with ssslib enabled components**
**shellTest**
**execute a command**
**httpTest**
**ie. app testing web interfaces (a’la Globus, etc)**
**tcpipTest**
**raw socket via tcpip transmission.**

## APITEST supports Regular Expressions

**Regular expression matching in tests**

**Sent to service-directory:****<get-location><location component=“*”/></get-location>**** **
**Expected Buffer:****<locations>****     <locations component=“service-directory”/>****     <locations component=“event-manager”/>****</locations>**
-  **Regular Expression:****<recvbuf type=“REGEXP”> <locations>(\s*<location\s+component=“\S*”\s*/>\s*)+</locations>****</recvbuf>**

**Notes:**
some example regular expression encoding for how to validate a response that have some or a lot of variant information.  
Exact matching simply isn’t flexible enough.

## Creating New Test Types is Easy

**A simple test that will always pass:**
- class passTest(Test):	__attrfields__=[‘name’]	typemap = {‘dependencies’:TODependencies}
- 	def setup(self):		pass
- 	def execute(self, scratch):		self.expect[‘foo’] = [ (‘REGEXP’, ‘a’) ]		self.response[‘foo’] = ‘a’

**Notes:**
Code highlighted in BLUE is templated stuff.  (would be required for all new test types).  The only *new* code a user really
would need to add are the two BLACK lines.

This slide illustrates how little one needs to create a new test type.
In this case, the test type always passes because expect the expect and response data
entries are identical.
The ‘scratch’ parameter to execute() can be thought of as a whiteboard.  It’s a dictionary
type that is passed into execute() for every test.  This can be used to pass information from 
one test to another.

## Matching and Aggregation

**An individual test can be executed many times in a sequence.**
**PASS/FAIL can be determined based on the percent of runs that matched.**
**Percent Match can be specified as a range as well.**
**Expected result is specified as a regular expression (REGEX) or a string for exact matching (TXTSTR)**
**Notation:**
**M****[min:max]**** - Percent matching.**
**min/max = bounds on % of tests where actual and expected results match.**
**If the actual number of tests is within the range specified the test will PASS, otherwise it will FAIL.**

**Notes:**
M[:0] = PASS assigned if a test failed 100% of its runs.  (ie: PASS only if 0% to 0% of runs matched expected output)
M[100:] = PASS assigned iff a test PASSED 100% of its runs.
M[30:60] = PASS assigned iff a test passed between 30% and 60% of its runs.

## Test Dependencies 

- T iff A[40:90]OR B[0:0]

- M[0:0]

- A

- B

- ?

- M[40:90]

- 

- T iff (A[100:]  B[90:]) C[:]

- M[40:90] : >= 40% and <= 90% of test runs matched

**Notes:**
Example of some dependencies.  

The LHS example could be read as “Execute test T if A has executed and passed between 40% and 90% of its runs OR if B has executed and passed 0% of its runs”

The RHS example could be read “Execute test T if all of A and at least 90% of B passed OR if C executed”

In all cases, a dependency will only attempt to run AFTER all cases have executed (or attempted execution).

## An Example Dependency

- (A[100:]  ((B[100:100]  C)  D[:0] ))

**<dependencies>**
**    ****<AND>**
**        ****<dependency name=’A' minPctMatch=’100'/>**
**        ****<OR>**
**            ****<AND>**
**                ****<dependency name=’B' minPctMatch=’100'/>**
**                ****<dependency name=’C'/>**
**            ****</AND>**
**            ****<dependency name=’D’ maxPctMatch=‘0’/>**
**        ****</OR>**
**    ****</AND>**
**</dependencies>**

## An Example Test Sequence

- reset daemon

- test daemon

- test other stuff

- M[:]

- M[:]

- M[:30]

## Standard Test Metadata Attributes

| Attribute | Type | Req’d | Default | Desc |
| --- | --- | --- | --- | --- |
| name | string | YES |  | name of test |
| numReps | integer | NO | 1 | number of times to execute test |
| minPctMatch | float | NO | 0.0 | min % of repetitions that must match for test to pass |
| maxPctMatch | float | NO | 100.0 | max % of repetitions that must match for test to pass |
| preDelay | float | NO | 0.0 | delay in seconds prior to executing test |
| postDelay | float | NO | 0.0 | delay in seconds after executing test before continuing to next test. |
| iterDelay | float | NO | 0.0 | delay in seconds between tests during loop (only used if numReps > 1) |
| onMismatch | string | NO | CONTINUE | If a test fails to match, what do we do?
CONTINUE,BREAK,HALT |

**Notes:**
Custom tests can have their own attributes available, for instance, shell tests have an attribute for the command line argument,
tests for ssslib specify the destination (ie. service directory component name) for the message.  A tcpip test can specify the 
URL/port for a message to be delivered to, etc.  This list merely is the set of attributes that APItest understands.

## Example Scripts

**A simple shell execution test:**
**   ****<shellTest  name=‘test1’ numReps=‘1’ preDelay=‘5’ postDelay=‘5.3’ ****                   command=‘ls -ltr’/>**
**Test with a dependency and stdout matching:**
**   ****<shellTest name=‘test2’ command=‘apitest.py --test’>****   <output format=‘REGEXP’ type=‘stdout’>.*stdout.*</output>****   <dependencies>****      <dependency name=‘test1’ minPctMatch=‘100.0’/>****   </dependencies>****</shellTest>**

## APItest Output

**iterations****	****test name****	****% matched****	****Pass/Fail****	****message**
**----------****	****---------****	****---------****	****---------****	****----------**
**[1 of 1]****	****	****A         100.00%****	****PASS     ****	**
**[1 of 1]****	****	****K         100.00%****	****FAIL     ****	****m[0.0% : 0.0%]**
**[1 of 1]****	****	****J           0.00%****	****FAIL     ****	****m[90.0% : 90.0%]**
**[5 of 5]****	****	****M         100.00%****	****PASS     ****	**
**[1 of 1]****	****	****L         100.00%****	****FAIL     ****	****m[0.0% : 0.0%]**
**[1 of 1]****	****	****N         100.00%****	****PASS**
**     ****	**
**[0 of 1]****	****	****T         DEPENDENCY FAILURE(S)**
**	****	****	****F expected [0.0% : 90.0%], got 100.0**
**	****	****	****J expected [90.0% : 100.0%], got 0.0**
**[0 of 1]****	****	****R         DEPENDENCY FAILURE(S)**
**	****	****	****J expected [90.0% : 100.0%], got 0.0**
**	****	****	****K expected [0.0% : 90.0%], got 100.0**
**[1 of 1]****	****	****S         100.00%****	****PASS     ****	**
**[1 of 1]****	****	****U1        100.00%****	****PASS     ****	**
**[1 of 1]****	****	****U2        100.00%****	****PASS     ****	**
**[0 of 1]****	****	****U3        DEPENDENCY FAILURE(S)**
**	****	****	****N expected [0.0% : 90.0%], got 100.0**
**	****	****	****S expected [0.0% : 90.0%], got 100.0**
**[1 of 1]****	****	****U4        100.00%****	****PASS**

## sssTest outputs from Chiba City

**iterations   test name****	****% matched****	**** Pass/Fail   message**
**----------   ---------****	****---------****	**** ---------   ----------**
**[1 of 1]****	****   add-location****	****100.00%****	**** PASS      **
**[1 of 1]****	****   QuerySDComps****	****100.00%****	**** PASS     **
**[1 of 1]****	****   QuerySDHost****	****100.00%****	**** PASS     **
**[1 of 1]****	****   QuerySDProtocol ****	****100.00%****	**** PASS     **
**[1 of 1]****	****   QuerySDPort****	****100.00%****	**** PASS     **
**[1 of 1]****	****   del-location****	****100.00%****	**** PASS     **
**[1 of 1]****	****   val-removal****	****100.00%****	**** PASS     **

**iterations   test name       % matched****	**** Pass/Fail   message**
**----------   ---------       ---------****	**** ---------   ----------**
**[1 of 1]****	****   sss-getproto    100.00%****	**** PASS     ****	**
**[1 of 1]****	****   sss-getport     100.00%****	**** PASS     ****	**
**[1 of 1]****	****   sss-gethost     100.00%****	**** PASS     ****	**
**[1 of 1]****	****   sss-getcomp     100.00%****	**** PASS     ****	**
**[1 of 1]****	****   sss-getproto    100.00%****	**** PASS     ****	**
**[1 of 1]****	****   sss-getport     100.00%****	**** PASS     ****	**
**[1 of 1]****	****   sss-gethost     100.00%****	**** PASS     ****	**
**[1 of 1]****	****   sss-getcomp     100.00%****	**** PASS     ****	**

**Notes:**
This slide shows output from actual runs from tests on Chiba City, querying the service directory component.

## Release Testing...

## Tales from Cplant Release Testing

**Methodical execution of production jobs and 3rd Party benchmarks to identify system instabilities, enabling them to be resolved. Ie:**
**Rapid job turnover rate (caused mismatches between scheduler and allocator)**
**Heavy io (I/O which passes through launch node process instead of directly to ENFS “yod-io”)**
**Wrapping above codes into Ctest framework to enable portable compile, launch, and analysis of synthetic workloads**

**Notes:**
This slide is from Jon Stearley (jrstear@sandia.gov) relating some of his experiences as release manager of CPlant.
the 3 sub-bullets are some example problems that turned out to cause the system software problems when users
did these things.  (allowing PBS to launch scripts enables much of this)

Ctest framework is under development for maybe the last 2-3 weeks by Jon.  It’s being used for simulated workload 
testing and is specifically designed to launch many jobs at specific times/intervals via the batch system.

## Ctest

**Extension of Mike Carifio’s work**
**Presented at the SciDAC meeting in Houston during Fall of 2002**
**Make structure that holds a suite of independent applications.**
**Tools to launch as a reproducible workload.**
**Goal: 30 users and 60 concurrent apps**

## Sample Load Profile on CPlant

**Notes:**
data collected by Jon Stearley, illustrates actual job profiles on CPlant.  This gives an idea of a workload over time on a production machine that runs ASCI applications.
It’s noteworthy that there are many jobs that run for 10+ days continually.

## Issue Tracking

**SNL uses a program called RT**
**Centralized repository for issue tracking helps give an overall picture of what problems are.**
**Helps give summary of progress.**
**Bugzilla is on the SciDAC SSS website**
_**[http://bugzilla.mcs.anl.gov/scidac-sss/](http://bugzilla.mcs.anl.gov/scidac-sss/)**_
**Who’s using it?**

## Status Daemon ...

## Status Daemon

**Highly configurable monitoring infrastructure for clusters.**
**Does not need to run daemon on the node you are monitoring.**
**XML configurable**
**Web interface**
**“****Cluster Aware”**
**Used on CPlant production clusters**
**James Laros (jhlaros@sandia.gov)**

**Notes:**
The main difference between this tool and other monitoring tools is that the status daemon does
not have to be running on the node it is collecting data from.  In the case of CPlant, status_daemon
does not run on the compute nodes, but rather ‘leaders’, giving more cycles to user applications
for higher performance on thousands of processors.

The primary contact for CIT and the Cstatus suite is James (Jim) Laros (jhlaros@sandia.gov)

## Status Daemon Communication

**Notes:**
Graphical representation of where the status daemon will run.  The blue box represents the administration node (1 per cluster)
The Green boxes represent the ‘leader’ nodes, and red boxes represent compute nodes.  The main idea of this slide is that 
the status daemon can be run on the admin and leader nodes, collect information remotely from compute nodes with minimal
disruption of compute processes, and merge the information back up to the admin node where it is collected and can be displayed
in a readable manner to the administrator.
Also, changes to the configuration are filtered down from the admin node to the leaders if you want to view different data on the 
machine.

## Slide 26

**Notes:**
Just some slides on the web interface for the status daemon, demonstrating the information that can be collected and the presentation style.

## Slide 27

**Notes:**
more demonstration of the RRDB of production data on CPlant.

## Slide 28

## Slide 29

**Notes:**
example output from exeucting status daemon on ROSS

## Slide 30

**Notes:**
example XML control file for status daemon.

## Slide 31

## Summary

**New Hire**
**Ron Oldfield**
**APItest functionality and flexibility increases**
**Release testing experience**
**Status Daemon**

## Plans

**APItest**
**User / Programmer Manuals**
**User Interface**
**GUI?  HTTP?**
**daemon mode for parallel testing mode**
**DB Connectivity**
**Test Development**
**ssslib event tests**
**HTTPtest work**
**ptlTest (SNL)**
**SWP Integration**
**Port SWP to Chiba for SC2003?**

**Notes:**
Can we port the Simulated Workload Project code to Chiba, since it’s running SSS? 
Can’t offer guarantees prior to SC2003, but the SWP framework is set up under gmake
and isn’t really designed to be a stress test, but a usage test.  Many apps in the Sandia
repository are sandia-specific and while the codes themselves aren’t all that bad, many 
require large libraries to link against which may or may not be ported easily to a new
architecture/environment. 

## Questions?

## END OF PRESENTATION

## APITEST Component Architecture

**Notes:**
This shows the general component architecture of apitest as it stands today.

## Encoding XML into XML CDATA

- <ssslibTest>
-     <name>test1</name>
-     <destination>sd</destination>
-     <sendbuf>
**<get-location>**
**<location component='*'/>**
**</get-location>**
-     </sendbuf>
- </ssslibTest>

- XML parsers will interpret the **“<“** character as a newElement, NOT as a character buffer.  So,we must trickour SAX parse by using **&lt;** instead of **<**

**Notes:**
informational slide:
putting a “<“ into CDATA will cause problems, but using &lt; gets past the parser and will be correctly interpreted into a “<“ by the application.