# Presentation: 378936

## Current Issues

March 9, 2000, 1:30pm

Jim Tomkins - SNL
Session Organizer

John Noe - SNL
Susan Coghlan - LANL
Greg Lindahl - UVA

**March 9, 2000, 1:30pm**
**Jim Tomkins - SNL**
**Session Organizer**
**John Noe - SNL**
**Susan Coghlan - LANL**
**Greg Lindahl - UVA**

**Integration and Testing Technology**

## Integration and Testing for Operations

**Clusters composed of readily available COTS parts are easy to assemble. ****(relatively)**
**Packaging varies from open shelf to custom cabinet rack mounted ala CplantTM.  More structure may help integration effort via subassembly testing and completeness verification. ****Are all the parts there?  Are they the correct parts?**
**First level integration is fitting the proposed system together on the floor.  ****Will it fit in the building?****  ****Can you transport pre-assembled units or must it be site assembled?**
**Burn-in testing (ie electrical and mechanical connectivity/control) doesn’t validate system health but is a necessary starting point.  ****Are all the parts working?**

## Integration and Testing

**But....Integration testing must be aimed at identifying problems at a system level:**
**bootp process (administration of system)**
**message passing (control network and data network)**
**file system access**
**compiler and library suitability**
**network stability and performance**
**Not all issues will be found, especially scaling surprises, but fundamental problems should be caught and design decisions rethought early in the procurement cycle.**

## Integration and Testing (cont.)

**Once the hardware is on the floor, the time for ****integration**** testing is long gone. At this point, we are doing ****validation**** testing.  ****Is the system doing what we expected?**
**Prototype systems of small numbers of components need to be thoroughly tested prior to a large order.**
**Point in fact - ****Ethernet switches versus hubs on Siberia CplantTM .****  Interaction of switches and XP1000 systems during power cycles and multiple boot attempts proved troublesome.**
**Post mortem analysis identified a reliable operation realm which is in place and working.  Diagnostic/control network is set at Fast (100Mb/s) and Half-duplex.**

## Integration and Testing (cont.)

**Prototype systems of small numbers of components need to be thoroughly tested prior to a large order.**
**Point in fact**** - GigEther interfaces for I/O.  Initial throughput at lower rate than 100Mb ethernet, 7 MB/s versus 10+ MB/s. **
**However, GigEther connections were more stable than the 100Mb ethernet connections.**
** ****Problems with 100Mb ethernet included adapter resets at high load.  Adapters would reset to 10Mb mode.**

## Integration and Testing (cont.)

**Point in fact**** - GigEther jumbo packets (9000 mtu vs 1500 mtu). Performance limited to about 30MB/s.  Subsequent testing pushed memory-to-memory stream rates to 100+MB/s.**
**Investigation conducted by Jim Schutt (SNL) revealed unusual aspects of Alteon 180 switch (VLAN and routing behavior) and dependencies associated with driver code, including method of loading the driver.  Details coming in a SAND report.**
**Point in fact**** - Nodes with two ethernet connections will reset the device number of the motherboard interface to be the second logical device.**
**Point in fact**** - Cisco 8540 switches did not support jumbo packets as advertised.**
**Point in fact**** - Full Linux operating system sometimes presents problems repeating timing studies.....daemons running at odd times.  **

## Integration and Testing (conclusion)

**So my point in fact:**
**Integration testing needs to be done as part of the design cycle, not after the fact.  This testing includes the administrative scripts that control configuration of the system, the operating system build, message passing libraries, batch job controller, compilers, loaders, file systems, diagnostic systems, etc.**
**Testing of the integrated system must ****validate**** the hardware order and demonstrate system continuity. Does the system “hang together” as one unit, boot first time everytime, give a hint at what is broken when it breaks, and run user level programs well and consistently?**
**This same level of testing is needed as new system releases are introduced to ensure reliability, availability, and stability for user level codes.**