In this Issue... 

Impulsive Nerve Agents 

Simulating Real Flight Using HPC 

HAWK - The Newest Addition 

A Scrubber Scans for Aged Files 

How to Track Down Dead Jobs 

CCAC, Not Just Another

Acronym

Near Space, The Next 

Frontier

The Path to CCE Leads 

to Accessibility 
Table of Contents Powering

Tomorrow’s

Solutions

Director’s Desk .......................................................................................................... 1

Features

Protection and Cure in the Face of Nerve Agent Exposure................................... 2

Research Reduces Airplane Tests with Increased Resolution of Simulations....... 5

Technology 

Technology Insertion 2007 Adds Power for Users................................................. 7

User Focus

Workspace Scrubber: Friend or Foe?.................................................................... 8

This Job is Dead!................................................................................................... 9

Baseline Configuration - A Maturing Initiative ...................................................... 10

The ezHPC Eases Users’ HPC Needs ................................................................ 12

News

Welcome to CCAC............................................................................................... 13

Human Interest

ASC MSRC Staff Receives Numerous Awards and Nominations ....................... 15

2006 Supercomputing Conference...................................................................... 16

Hails and Farewells ............................................................................................. 18

Recent Visitors..................................................................................................... 19

PET

MaXFlyer Near Space Vehicle............................................................................. 20

A Consistent Computational Environment ........................................................... 21

SIP High Productivity Tool .................................................................................. 24

Former PET Intern Wins Award ........................................................................... 24

Upcoming Conferences ....................................................................................... 25
Spring 2007 Wright Cycles 1

Director’s Desk Powering

Tomorrow’s

Solutions

Greetings from the Birthplace of Aviation: Dayton, Ohio. Our heads may be in the wild blue 

yonder, dreaming of the possibilities with our new SGI Altix 4700 online, but I assure you our 

feet are firmly grounded as we plan for the realities of providing over 16,000 processor-cores to 

our Defense Department’s valued scientists and engineers. 

Check out Lloyd Slonaker’s article on our latest addition, the 9216-core SGI Altix. We are 

working closely with SGI and the High Performance Computing Modernization Program 

(HPCMP) Office to make this behemoth both useful and usable at the earliest 

possible date. We are striving to assure high availability levels for the 

system and place it in an environment conducive to ease-of-use. 

Our users have been executing some amazing projects this year. Please review 

articles or work by Drs. Margaret Hurley, Scott Morton, and Hugh Thornburg 

for highlights regarding the development of compounds for protection and 

removal of nerve agents from the human body’s system, virtual flight testing, 

and design of an unmanned near-space vehicle. Although these represent a 

small fraction of the processing work being performed here everyday, they are 

an eclectic and astounding sample nonetheless! 

More down-to-earth are articles featuring helpful strategies for users, 

the HPCMP’s Baseline Configuration Project, and the recently stood-up 

Consolidated Customer Assistance Center (CCAC). All of these articles 

will be of interest to every user. If the articles leave you with questions or 

concerns, or ideas for improvement, I need and urge you to communicate with 

the authors. We value 2-way communication! 

My most down-to-earth preoccupation continues to be that of facilities. 

My dream is of a gleaming white expanse with bountiful space, power, and 

cold air, but alas there is no such tundra in Dayton, nor does the Military 

Construction Program look likely to build such a place for some time. Instead, 

we must keep our current home in good repair, feed it more power and 

cooling, and build robust utilities and infrastructure within the constraints 

we’ve been handed. 

So we’ll continue to provide and maintain a great roost so that EAGLE, 

FALCON, and HAWK can soar and “[slip] the surly bonds of earth”!1 

1

From the poem “High Flight” by John Gillespie Magee, Jr.

Steven J. Wourms 

Director, ASC MSRC 
Features e g 

o ’ 

Powering

Tomorrow’s o ’ 

Solutions o t

BY GARY SIVAK AND DINAH LUNEKE

Dateline March 20, 1995, Tokyo, Japan. In an act of 

domestic terrorism, members of an obscure religious group, 

Aum Shinrikyo, perpetrated a Sarin gas attack on the Tokyo 

subway system, injuring thousands, and killing 12. 

What if a similar attack were to occur against a group of 

our warfighters in the mountains of Afghanistan today? 

The goal of this project is to facilitate the development of 

compounds that can be administered to our warfighters for 

called bioscavengers, to sweep up nerve agents from 

the body’s systems over time. According to Dr. Hurley, 

“[the team] wants to develop a soup-to-nuts model of this 

enzyme in order to understand binding to and unbinding 

from cholinesterases, how to flush nerve agents out of the 

human body’s systems, and what compounds to administer 

to protect soldiers.” 

Technical Details 

Acetylcholine (ACh) is a neurotransmitter that transmits 

impulses from the human body’s nerves to the muscles. 

After the muscle contracts, the body uses AChE to 

break down ACh, thereby instructing the muscle to 

stop contracting and to relax. When nerve agents bind 

to and irreversibly inhibit the action of the enzyme, the 

concentration of this neurotransmitter builds up, the nerves 

remain polarized, the muscles contract fiercely, and usually 

suffocate the victim by preventing breathing and heart 

Protection and Cure in the Face of Nerve 

Agent Exposure 

This Challenge project, “Defense Against Chemical

Warfare Agents (CWAs) and Toxic Industrial Chemicals

(TICs): Filtration, Prophylaxis and Therapeutics,” was

run at the ASC MSRC by Dr. Margaret Hurley, Army

Research Laboratory (ARL), Aberdeen, Maryland, and

collaborators Dr. Gerald Lushington and Dr. Jianxin Guo

of the University of Kansas. ASC MSRC HPC System

Utilization: 590,698 total hours

their protection and cure. While the primary emphasis of action.

this research focuses on military applications, there are very 

far-reaching connections; as Dr. 

Hurley pointed out, “As this 

could be of interest for anyone 

from emergency first-response 

teams to agricultural workers 

exposed to chemically similar 

pesticides.” 

Research Goals 

Through their computational 

chemistry studies at the ASC 

MSRC, Dr. Hurley and coworkers are investigating the 

many complex reactions by 

which nerve agents and other 

compounds bind with the 

enzyme acetylcholinesterase 

(AChE) within living systems. 

This project keeps progressing 

as new developments, 

refinements and discoveries are 

made. One goal is to analyze 

and help develop improved 

scavenger compounds, also Molecular dynamics simulations show sympathetic side chain flexion assisting nerve agent 

transport to the active site. 

2 Spring 2007 Wright Cycles 
PHE330

TRP84

HIS440

GLU327

SER200

Molecular Dynamics 

Quantum

Docking

Measured pKi 

Predicted pKi 

4 5 6 7 8 9 10

10

Definitions

Acetylcholine (ACh) - a neurotransmitter that mediates between nerves

and muscles. It allows electrical

impulses to transmit along the nerves

to the muscles. ACh is in the group

of compounds called organic acid

esters.

Acetylcholinesterase (AChE) - an

enzyme produced in the body that

breaks down the neurotransmitter

acetylcholine after it instructs the

muscle to contract.

Conformation - the allowed motions

within the pieces of a molecule, as

well as the shape and orientations the

pieces can assume, as constrained

by the bonds within the structure of a

molecule.

Nerve Agent - a lethal chemical

(organophosphate, liquid or gas) that

binds to the AChE enzyme, preventing it from breaking down a build-up

of ACh. Nerve agents disrupt the

mechanism by which nerves transfer

impulse messages to both organs

and muscles.

Reference: http://en.wikipedia.org/

wiki/Nerve_gas

QSAR - Quantitative StructureData at multiple time/length scales from theoretical modeling (docking, 

moleculardynamics,quantummechanics) are usedtostudy allaspects 

of transport and binding in cholinesterase systems. 

Activity Relationship. A chemical

statistical analysis technique.

Scavenger Compounds (also known

as Bioscavengers) - function in the

Although this research is predominately applied science/ 

chemistry, there is also a great deal of basic chemistry 

used. “The molecular interactions that hold the enzyme 

in its optimal conformation (shape) and promote catalysis 

have never been elucidated,” Dr. Hurley explained. 

“Therefore, toxicity prediction of nerve agents and antidote 

effectiveness can only be hypothesized by extrapolation 

from the behavior of familiar chemicals. A sound 

theoretical basis would allow reasonable predictions on 

compounds unrelated to those already studied.” 

DoD HPC resources enable the team to extend their study 

of AChE inhibition to three increasingly intensive levels of 

chemistry theory: 

• Virtual screening of inhibitor formulation (molecular 

docking) utilizes a Quantitative Structure-Activity 

Relationship (QSAR) database search. Pharmaceutical 

companies use this technique to screen candidates, and 

zero-in on likely active compounds for new drugs. 

blood, like a sponge, to absorb the

nerve agent before it can do its damage. They sweep up nerve agents

because they bind to the nerve

agents more strongly than the nerve

agents bind to AChE. Reference:

http://Nexiabiotech.com

• Analysis of competitive transport kinetics (rate 

equations) involves nerve agents and therapies reacting 

with AChE. This deals with transport to the active site 

on the enzyme, i.e., the molecular movement, rather 

than typical chemical reactions or bonds. 

• Prediction of reaction kinetics, thermodynamics and 

physical properties consists of fundamental modern 

chemistry using HPC and the equations of quantum 

mechanics. It predicts the geometries and energies 

(strengths) of the chemical bonds between AChE, nerve 

agents, and therapeutic agents. 

continued on next page 

Spring 2007 Wright Cycles 3
“In this research there exists so much activity occurring 

among the chemical constituents that you must study the 

effects on different time and length (distance or size) scales. 

The scales differ by many orders of magnitude,” according 

to Dr. Hurley. “This is a multiscale study, and one code 

cannot compute it all. This research is not an idiot-proof 

black box approach, results are emerging simultaneously 

from quantum chemistry, molecular dynamics, and 

molecular docking, and results from each scale help 

provide a reality check for what’s going on at the other 

scales.” 

Value to the DoD 

Dr. Hurley says that the beauty of this research involves 

saving money on lab experiments and tests. For example, 

her QSAR methods have demonstrated a high level of 

predictivity, yielding a high correlation value of 0.69 

between the calculated results and experimental values, for 

a 16 molecule test set. 1 “With other methods, you’re lucky 

to get a correlation value of 0.13,” according to Dr. Hurley. 

Additionally, a sound working model of AChE inhibition 

would also help speed the scavenger development process, 

References

1

 Multiscale Computational Studies of Cholinesterase Inhibition for 

Improved Therapy and Protection Against Nerve Agent Exposure 

Advancing the Front at ASC 

Dr. Hurley primarily uses the Commercial Off-The-Shelf 

(COTS) codes Gaussian and Amber on the SGI Origin 

3900 system at the ASC MSRC. “I like running at the ASC 

as their user services are great, and they do a great job with 

a completely seamless process,” Dr. Hurley remarked. 

“The ASC MSRC HPC systems are stable and provide the 

reasonable throughput that I need. Running on these HPC 

systems affords me a higher fidelity for my research. In 

computing using quantum mechanics, it is impossible to 

achieve this caliber of result without HPC. In the future, 

I expect more of the same level of quality service that I 

currently receive.” 

thereby better protecting our warfighters. 

For more information, contact the ASC MSRC at 

asc.hp.outreach@wpafb.af.mil.

User Tip!

Debug Queue

Check for errors in these areas:

 Syntax errors

 Environment variables

	 Successful copying of files

 Changing to the right directories

 Ensure mail messages are sent

More info: www.asc.hpc.mil/customer/userdocs User Tip!

Quick Reference Guides are readily

available on our website. Gain the most

from our HPC resources by accessing this

documentation designed to help you beĴer

understand how to use the resources of the

ASC MSRC.

www.asc.hpc.mil/customer

4 Spring 2007 Wright Cycles 
Spring 2007 Wright Cycles 5

BY GARY SIVAK AND DINAH LUNEKE

Project Identification 

Two HPCMP Challenge projects were run at the ASC 

MSRC by Dr. Scott Morton, Technical Manager of the 

Institute for HPC Applications in support of the Air 

Armament, AF Seek Eagle Office at Eglin Air Force 

Base. The first project, “Multidisciplinary Applications of 

Detached-Eddy Simulation to Separated Flows,” began 

in FY05, and the subsequent project, “High Resolution 

Simulation of Full Aircraft Control at Flight Reynolds 

Numbers,” started in FY06. 

Spinning out of Control 

You are the Program Manager for the F-89 Fighter Attack 

Group, a high visibility program, which is developing the 

F-89, a state-of-the-art super-stealthy attack fighter for the 

Air Force. Your Contracting and Finance Program Office 

informs you that the Air Force has just slashed your flight 

test budget by 70 percent. Next, you receive a call from 

flight test personnel at Edwards Air Force Base, informing 

you that, due to busy flight range schedules and overbookings, the number and time of your scheduled flight 

tests have been cutback by 80 percent! Will your F-89 

program crash and burn? This article outlines how Dr. Scott 

Morton of the Seek Eagle office and the high performance 

computers at the ASC MSRC came to the rescue. 

A Seeker of Excellence 

Dr. Morton envisions his work as virtual flight 

testing, which can make more effective use of or 

even reduce actual flight tests. His team’s simulations 

augment actual flight testing for stability and control 

to make sure the aircraft flies the way it is designed to 

fly. “You run through different maneuvers and develop 

a composite aircraft model by combining CFD with 6-

degrees of freedom motion, i.e., translation and rotation of 

the vehicle,” Dr. Morton said. “Current flight test capability 

is not sufficient for our future programs, and therefore a 

new simulation capability is essential.” 

Research Approach 

The work involves aspects of both application and theory 

and includes these important flight testing steps: 

1. Simulation Model Integrity. This is an attempt to repeat 

the same maneuver as performed in flight tests during 

aircraft certification for flight worthiness. “We are trying 

to develop new treatments of turbulence to improve 

simulation of aircraft with massively separated flows,” 

said Dr. Morton. “You can incorporate other disciplines, 

such as maneuverability or aeroelasticity to improve the 

fundamental physics models. Through these simulations, 

we want to understand specific characteristics of control for 

air vehicles doing complex maneuvers.” 

2. Minute Maneuvers. For simulations within the 

Research Reduces Airplane Tests with 

Increased Resolution of Simulations 

continued on next page 
operational envelope (or range of possibilities) of an 

aircraft, maneuvers are dissected into their various 

components to gauge the aircraft’s sensitivity or response to 

each individual component. This type of test is impossible 

to perform with an actual aircraft. For example, turning an 

aircraft away from its present course involves both rolling 

the aircraft into the turn with the stick and making the 

actual turn with the rudder controls. Individual maneuver 

components, such as these, are impossible to perform in the 

air, and therefore can only be tested through simulation. 

3. Out of Bounds. Simulations can go outside the flight 

envelope or expected flight scenarios into risky territory 

so that the pilot doesn’t have to go there. It is then known 

in advance that some maneuvers are not necessarily safe, 

before pilots would ever experience them and thus risk their 

lives. A good example of this would be an out-of-control 

spin.

Project Strategy 

“We are applying proven CFD techniques to what we’re 

calling ‘virtual flight test,’ but we’re also developing 

new maneuvers to improve the efficiency of flight - the 

theoretical side,” said Dr. Morton. 

The work does not eliminate but rather augments flight tests 

to reduce the number of tests needed and increases their 

payoff. For example, to certify a particular test point in the 

sky (safe operation at 400 knots), traditionally operations at 

350, 375, 400, 425 and 450 knots would need to be verified. 

With a CFD simulation, it is only necessary to verify that 

400 knots is safe and conduct one flight test instead of five. 

These HPC studies protect both the integrity of the aircraft 

and pilots’ lives, while giving the exact information needed 

by the test team. 

The Power of ASC 

Dr. Morton likes running at the ASC MSRC. “I have a good 

working relationship with the great support staff at the ASC 

MSRC,” Dr. Morton said. “I’ve been running Challenge 

projects at the ASC MSRC for more than six years and 

want to continue in that relationship.” 

Morton’s application of choice is the Computational 

Fluids Dynamics (CFD) program, Cobalt, which he runs 

on the FALCON Linux cluster. FALCON is well-suited 

to Cobalt since the application scales well. Cobalt has a 

super-linear scaling curve up to 4,000 processors, and he 

has been able to achieve that level of super-linearity. Cobalt 

can efficiently run the large, geometric grids representing 

large, complex, aircraft vehicles with stores (i.e., weapons, 

bombs, fuel tanks, and sensors). 

Value to the DoD 

Dr. Morton’s Challenge projects benefit the warfighter 

by reducing the time and money spent on flight tests. For 

example, the F-22 and F-35 programs have less money to 

spend on flight tests, so the difference is made up in high 

fidelity simulations which augment expensive testing. 

For more information, contact the ASC MSRC at 

asc.hp.outreach@wpafb.af.mil.

Aircraft CFD flow field representation 

6 Spring 2007 Wright Cycles 
Technology Powering

Tomorrow’s

Solutions

Technology Insertion 2007Adds Power for Users

BY LLOYD SLONAKER

Powering tomorrow’s solutions by providing high 

performance solutions for users just got a lot easier at 

the ASC MSRC. The integration effort has begun on the 

Technology Insertion 2007 (TI-07) upgrade for the Center. 

The High Performance Computing Modernization Program, 

in coordination with GSA, awarded a contract to SGI 

Federal, Inc. that will provide a new high performance 

computing system with three times the performance 

capabilities currently available at the ASC MSRC. This 

new SGI Altix 4700 will be named “HAWK,” in honor of 

the F-117 “Night Hawk.” The new SGI system will rank as 

one of the fastest computers in the DoD. 

The high performance supercomputer will be powered 

by 4,608 dual-core, 1.6 GHz Intel Itanium 2 processors 

for a total of 9,216 processor-cores, separated into 18 

nodes, with 512 processor-cores. The system has 20TB 

of shared memory and 440TB of usable disk space. There 

will be 16 nodes with 2 GB per core and 2 nodes with 4 

GB per core. Each node will have 500 cores available for 

user applications with 12 cores set aside for its operating 

system. The system will also have a 32-core Interactive 

Front End (IFE). This IFE is where users will compile 

and submit jobs for execution on the system. Because of 

the compact blade packaging, each node will consist of 4 

racks for a total of just 72 racks for the system. Because of 

the density of the new SGI system, air cooling the system 

would have been a major concern. However, this was 

solved by SGI providing a system with water-cooled doors. 

This state-of-the-art water-cooling technology will ease the 

burden of cooling the system by providing a more direct 

and efficient heat exchange methodology. 

The operating system will consist of the SUSE Linux 

Enterprise Server version 10 with SGI’s ProPack 

5 extensions. With the shared-memory NUMAflex 

architecture and NUMAlink interconnect, the system will 

scale application performance for up to 500 processors, 

all working together in a cache-coherent manner. The 

system will utilize CXFS for the 440TB of disk space. 

Architecturally, the HAWK system will be similar to that 

of the ASC MSRC’s SGI Altix Bx2 “EAGLE” system, 

procured during TI-05. Therefore, it was decided that 

HAWK’s workload management will use Platform LSF 

Our new system packs 9,216 Intel Itanium 2 Processors, 

20TB of memory and 440TB of usable disk space 

version 6.2. This will allow for a consistent user experience 

across the Altix systems at the ASC MSRC. Based on 

a 64-bit Linux operating environment, the system will 

provide optimal resource usage for demanding technical 

applications.

Because of the water-cooled doors and additional power 

requirements, the ASC facility has already started updating 

and upgrading the building power and cooling capabilities. 

Additional power panels and power distribution units are 

being added to handle the additional electrical loads. New 

water pipes will be installed under the raised computer floor 

to feed the water-cooled doors. The new system is expected 

to arrive in May and be available for limited production in 

the summer, with full production targeted for the first of 

October 2007. 

For more information, contact the ASC MSRC at 

asc.hp.outreach@wpafb.af.mil.

Spring 2007 Wright Cycles 7
User Focus Powering

Tomorrow’s

Solutions

Workspace Scrubber: Friend or Foe?

B Y R EBECCA L ANGDON

When users think of how the scrubber removes their 

workspace files, friendly is probably not one of the words 

they would choose to describe it. Yet the scrubber is vital to 

ASC MSRC users as it ensures that they have enough space 

to store their batch job files. 

Of the file systems available to ASC MSRC users, 

workspace is ideal for executing programs because it is 

local to each HPC system. Workspace provides users with 

fast access to files and does not impose quota restrictions. 

However, workspace is only intended for temporary 

file storage; users’ home ($HOME) and archive 

($ARC) directories provide permanent file storage 

on networked file systems. 

The temporary nature of workspace is 

maintained by the scrubber, a program 

that removes aged files and directories 

directory ($WRK) or in the batch job’s workspace directory 

($WORK_DIR). When a batch job begins running, a job 

directory (<job number>.<system>) is created in $WRK 

and is accessible from the batch job script via the $WORK_ 

DIR variable. The scrubber receives a report of all running 

jobs from the job scheduler and simply ignores their 

from workspace daily. Since workspace Illustration of file structure 

is a limited resource shared by system 

users, the scrubber is necessary to 

ensure that sufficient disk space is 

always available. The scrubber 

plays conflicting roles of foe and 

friend when it deletes a user’s files 

to free up more space for the user 

community. 

In general, the scrubber removes files 

and empty directories that are N days old.1 The

value of N must be large enough for workspace to 

be useful yet small enough to prevent workspace 

from becoming full. If workspace disk usage is too 

high after the scrubber has removed all content that is N 

days old, the scrubber will reduce the value of N by one 

day and run again. For this reason, we encourage users 

to be proactive by removing their workspace content, 

when finished, to ensure that enough workspace remains 

available to all users. 

If the scrubber removes files that are N days old, what 

happens to the files of a batch job that runs for more 

than N days? The answer is that it depends on whether 

the job files are stored directly in the user’s workspace 

$WORK_DIR directories 

until they have finished. 

Thus, users should store 

their batch job files in 

$WORK_DIR to protect them from the 

scrubber while the job is running. 

The classification of the scrubber as friend or foe is up 

to each user. By following the recommendations in this 

article, users can avoid the wrath of the scrubber altogether, 

but they will continue to benefit from its mission to ensure 

that enough workspace remains available to system users at 

all times. 

For more information, contact the ASC MSRC at asc. 

hp.outreach@wpafb.af.mil, or visit www.asc.hpc.mil/ 

customer/userdocs/ 1 At the time this article was written, N was five for all ASC MSRC HPC systems; the current values of N for each system are posted on the Workspace

Policy page on the ASC MSRC website (http://www.asc.hpc.mil/overall/policy_procedure/policies/wrkspace_pol.php).

8 Spring 2007 Wright Cycles 
This Job is Dead! 

BY JAY BLAIR AND DR. SCOTT KAJIHARA

There is a famous Monty Python skit where an unhappy 

customer, played by John Cleese, tries to return a very 

obviously dead parrot to a dodgy pet store shopkeeper 

played by Michael Palin. The skit degenerates into a 

lengthy disagreement on what constitutes being “dead.” 

The line that resonates throughout the skit is “this parrot is 

dead!” While not the first time that Monty Python has been 

referenced in a technical article on HPC, this particular 

reference is timely in that HPC support staff have, on more 

than one occasion, heard: “this job is dead.” The common 

thread between the HPC user’s job and the parrot is what 

constitutes being dead and how one tests for this condition. 

Nothing is more frustrating to a user than to submit a job 

to an HPC machine only to find that, after a period of time, 

the job has either not completed successfully, has taken 

longer to complete than necessary or worse, seems to have 

disappeared completely. The resolution process usually 

starts with a call to the HPC Help Desk support staff. 

Thankfully, the conversation with the Help Desk does not 

mirror the exchange that Cleese and Palin had in their skit. 

For the most part, users prefer to have tools, methodologies 

and answers at their disposal in order to correct any 

encountered difficulties with their work flow so they can 

get back to the task at hand. An informed user can often 

get to the root of the problem and become a partner in the 

resolution, instead of just a customer, if support is required. 

Fortunately, the ASC MSRC uses Platform Computing’s 

LSF program for workflow, and the commands to diagnose 

job issues are available and common to all of the HPC 

machines at the ASC MSRC. 

There are two basic commands used by the informed 

user or ASC MSRC staff member to determine what has 

happened to any particular LSF job: bjobs and bhist.

The bjobs command is used to gather information about 

batch jobs in LSF. The syntax for a specific job is bjobs

–l <jobid>, where the jobid is the unique LSF job 

identifier supplied to the user at submission. The bhist

command is used to display historical information on a 

job. The syntax for a specific job is bhist -l -n 0 

<jobid>.

These commands can help decipher a number of common 

user problems: jobs not starting, poor turn-around times, 

and jobs that seem to complete with no output. Both of 

the first two problems can be traced to jobs not starting or 

starting with unknown delays that impact the turn around 

times.

In the section PENDING REASONS from the output of 

bjobs -l <job-id> we can see: 

The queue has reached its job slot limit;

Job slot limit reached: N hosts; Not enough

free cpus to satisfy job requirements: N

hosts;

This translates into a shortage of processors available to 

start the user’s job. 

Resource (CPH) limit defined cluster-wide has

been reached; 

This translates into the user has/would have exceeded CPUhour limit on system. 

Job’s requirements for reserving resource

(foobar) not satisfied: N hosts; 

This translates into a job requiring a license or resource 

[foobar] that is not available in sufficient quantity. 

The issue of a job not pending or running with no output, 

the output for bhist -l -n 0 <job-id> shows us: 

Unknown; unable to reach the execution host; 

This shows that the execution host became unavailable or 

crashed and that the job should be resubmitted. 

These commands should be utilized whenever users require 

more insight into what is happening with their jobs. If 

the user is unable to determine the next steps from here, 

sending the output of these commands allows the service 

technician a head start in tracing down more information. 

At the very least, users will be better informed on whether 

or not the job is indeed “dead.” 

For more information, contact the ASC MSRC at 

asc.hp.outreach@wpafb.af.mil.

Spring 2007 Wright Cycles 9
Baseline Configuration - A Maturing Initiative 

BY JEFF GRAHAM

As highlighted in last year’s Wright Cycles, Baseline 

Configuration addresses the fact that customers of the 

DoD’s HPCMP continue to struggle with many different 

paradigms implemented across the centers and the systems. 

Differences in queuing systems, web-based information, 

policies, and procedures create difficulty when customers 

try to move from center-to-center and from system-tosystem based on availability of allocations. Service/Agency 

Approval Authorities (S/AAAs) have struggled for years in 

matching user needs and particular preferences, whether it 

is software, scripting, or types of queues. 

Check out the website at http://www.asc.hpc.mil/

consolidated/bc/ to review the completed projects – all 

16 of them! There are 11 additional projects currently 

in process to become guidelines in the near future at the 

six centers participating in the effort. By checking the 

Compliance Matrix, it is easy to realize why efforts persist 

to convert the fill-in-the-matrix into an automated process. 

Users have the ability to provide inputs. The team dedicates 

a great amount of time towards reviewing and readdressing 

customer inputs received though the user input form and 

via email. The team lead and other key members have 

presented the initiative to the Center Board of Directors, at 

PET Tech Reviews, at S/AAA and UAG meetings, during 

user interviews for Requirements Team visits, and to 

interested external parties including NASA and the HighEnd Computing Implementation Working Group. All the 

while, the team has adjusted their workload to handle the 

sustainment of completed projects by keeping them current 

and applicable to user productivity. The group is busier 

than ever before. 

As the reach of the Baseline effort extends, the program 

looks to the policies developed as part of the Baseline to 

help assure practices remain common across centers in 

times of budget uncertainties. This will keep our customers 

productive even as the staff across the centers is stretched 

thinner as consolidation activities replace long-standing 

levels of service. 

What is next for the team? The answer is more of the 

same including more user input, more weekly telecons, 

more maintenance of the website, (with support from the 

10 Spring 2007 Wright Cycles 

amazing ASC web development team), more projects 

turning into policies, and more work for a dedicated group 

of individuals known as the Baseline Configuration Team. 

My thanks goes to this team of dedicated professionals! 

Baseline Configuration Team 

Jeff Graham - Team 

Lead

ASC MSRC 

Sheryl Gage Instrumental

Cal Kirchhof Instrumental

Art Orth Instrumental

Dr. Aram Kevorkian - 

User Interface Manager 

UAG-SPAWAR 

Steve Finn UAG-DTRA 

Dr. Dave Cronk PET CE 

Center Representatives 

Mike Knowles ARL

Larry Haas ASC

David Sanders ERDC 

Christine Cuicchi NAVO 

Tom Logan ARSC

Mitch Murphy MHPCC 

User Tip!

To see an up-to-date matrix of

Baseline Configuration Project 

status, check out the website at:

www.asc.hpc.mil/consolidated/bc

For more information, contact the ASC MSRC at 

asc.hp.outreach@wpafb.af.mil.
The tables below show the projects, as identified by the Baseline Configuration Team, and their status 

Projects in Progress 

Project Number Policy Topics 

FY06-07 Common Web Documentation 

FY06-08 Common Policy for the Protection of User Data 

FY06-11 Commonality in Announcing Changes 

FY06-12 Rollback Policy 

FY06-13 Common Interactive Access 

FY06-15 Sample Code Repository 

FY06-16 Test Suite for Communication Libraries 

FY06-17 Use of Modules for Accessing Multiple 

Versions of Software 

FY06-18 Common System Status Monitoring 

FY07-02 Common Performance and Profiling Tools 

FY07-03 Common Set of Open Source Compilers 

Completed Projects 

Project Number Policy Topics POCs

FY05-01 Multiple-Version Software Mike Knowles 

FY05-02 Minimum Scratch Space 

Retention Time 

Mike McCraney 

FY05-03 Kerberos Ticket Life Larry Haas 

FY05-04 Environmental Variables Tom Logan 

FY05-05 Queue Names David Sanders 

FY05-06 Login Shells John Skinner 

FY06-01 Open Source Math Libraries Dr. Dave Cronk 

FY06-02 Kerberos/SSH Access John Skinner 

FY06-03 Subproject Allocation and 

Usage Command 

David Sanders 

FY06-04 Editors and Scripting Tools Mitch Murphy 

FY06-05 Debuggers Tom Logan 

FY06-06 Allocation Expiration Notice David Sanders 

FY06-09 Archival Retrieval Mike Knowles 

FY06-10 Standardized User Notices Mitch Murphy 

FY06-14 Cross-Center File Transfers Tom Logan 

FY06-19 Open Source Utilities Jeff Gosciniak 

Spring 2007 Wright Cycles 11
12 Spring 2007 Wright Cycles 

ezHPC interface provided by ERDC MSRC 

The ezHPC Eases Users’ HPC Needs 

BY DELBERT HICKMAN

When this new tool was developed by the 

ERDC MSRC for HPCMP users, the ASC 

MSRC quickly realized the potential for helping 

its users gain easy access to this set of secure 

and robust tools for accessing HPC systems. 

Expected benefits include: 

1. Access to HPC resources for users 

unfamiliar with the HPC environment and 

batch processing commands 

2. Development of new customized 

interfaces for specific applications and for 

processing paradigms, such as parameter or 

sensitivity analyses 

3. Availability of a graphical interface to 

batch job submission and management for a 

variety of user platforms 

Access the following websites to learn more 

about ezHPC and its related architecture: 

Architectural info: 

https://www.uit.hpc.mil/UIT/documentation.htm 

Help documentation: 

https://dolomite.erdc.hpc.mil/EZHPCWebClient/ 

ezHPCHelp.htm

ezHPC tool access: 

https://solutionhpc.erdc.hpc.mil/EZHPCWebClient/

To gain additional information on this new user 

tool, send your inquiries directly to solutionHPC_ 

support@erdc.hpc.mil. 

User Tip!

A list of user tools along with 

instructions for how to access 

are listed on the ASC MSRC 

user website.

https://user.asc.hpc.mil/
Powering

Tomorrow’s News Solutions

Welcome to CCAC 

BY MARIA ZIMMER AND DAN SCHORNAK

Launch of the Consolidated Customer Assistance Center 

The ASC MSRC is pleased to announce the launch of 

the Consolidated Customer Assistance Center (CCAC). 

This new customer support service combines the essential 

help desk functions across the DoD High Performance 

Computing Modernization Program (HPCMP) into one 

organization, dedicated to providing uniform, high quality 

customer support for all users of the HPCMP’s high 

performance computing resources. 

Managed through the ASC MSRC, CCAC will support 

users who have project accounts at any of the six 

participating shared resource centers within the HPCMP. 

The participating centers include the ASC MSRC, 

Army Research Laboratory (ARL) MSRC, Engineering 

Research and Development Center (ERDC) MSRC, 

Naval Oceanographic Office (NAVO) MSRC, Arctic 

Region Supercomputing Center (ARSC), and Maui High 

Performance Computing Center (MHPCC). 

This effort is one of several consolidation initiatives being 

performed within the DoD HPCMP. In addition to help 

desk services, CCAC will also provide, for a subset of 

participating centers, application management support for 

applications identified as part of the Consolidated Software 

Initiative (CSI) and open source codes identified as part of 

the Baseline Configuration Initiative (BCI). This support 

will broaden the offering of these applications across the 

Program, making it less difficult for users to move between 

shared resource centers as their processing requirements 

change or grow. 

The new Consolidated Customer Assistance Center, located at ASC MSRC, will serve six 

participating shared resource centers. 

continued on next page 

Spring 2007 Wright Cycles 13
CCAC Contact Information

Website: www.ccac.hpc.mil

Email: 

 Help Desk - help@ccac.hpc.mil

 Accounts - accounts@ccac.hpc.mil

Phone:

 1-877-CCAC-039 (toll free)

 937-255-0679 (local)

( 2222 )

14 Spring 2007 Wright Cycles 

What Users Should Know 

At the time of this printing, some of the participating 

centers are in the process of transitioning over to CCAC 

support services. 

Users need only to remember one toll-free phone 

number (1-877-CCAC-039) to make inquiries 

and request service support. CCAC offers 

extended service hours, 0800 to 2300 

Eastern Time every business day, which 

excludes weekends and federally 

observed holidays. Participating 

shared resource centers will 

continue to offer their same afterhours support outside of CCAC 

hours of operation. 

Specific site documentation 

for all participating centers 

will also be just a “click” 

away. CCAC will offer a 

web-based service that will 

initially provide links to each 

center’s user and system 

documentation. Since access 

to this information will be 

centrally located, users will 

only have to bookmark 

the CCAC website to 

keep abreast of the most 

current documentation 

available from each of the 

participating centers. 

Users may also use 

electronic-based service 

request systems to send 

emails to the CCAC Help 

Desk or submit their requests 

through the CCAC website. 

With CCAC’s extended hours, the processing of these 

requests will be handled efficiently and timely. 

Shared Resource Center Transition Plans 

Changes to existing services can be difficult for center 

support staff, but also equally difficult and challenging to 

users affected by these changes. To manage this, a plan 

was designed to stagger, in an orderly way, the transition of 

each center to CCAC support over a period of time. Care 

is being given to both the center and the center’s users 

in standing up these significant and beneficial services 

by ensuring issues are quickly addressed. As part of the 

transition process, affected users are being kept informed 

regarding the support contact changes and the timetable of 

these efforts. 

Going Beyond Today 

Once all six participating shared resource centers have 

transitioned to CCAC services, additional improvements 

and enhancements will be coordinated and rolled out 

over time. Current plans are to evolve several of the 

initial support services and tools that CCAC offers. Users 

are welcome to share their thoughts, observations, and 

suggestions with the CCAC team via the website, email 

addresses, or phone numbers listed below. This input will 

be critical in developing and making available the right 

service support tools and resources to users throughout 

the HPCMP. The ASC MSRC welcomes the opportunity 

and privilege of standing up the Consolidated Customer 

Assistance Center and is excited to provide this new 

support service to the DoD HPCMP community. 
Powering

Tomorrow’s Human Interest Solutions

ASC MSRC Staff Receives Numerous 

Awards and Nominations 

ASC MSRC staff receives recognition for their continuing commitment to excellence: 

The ASC Quarterly Award recognizes outstanding HQ 

ASC Air Force civilian employees and includes evaluations 

based on the accomplishments of job performance in the 

Primary Duty. Other factors considered for the award may 

include significant self-development, as well as Base/ 

Community Involvement. 

ASC 3rd Quarter 2006 Individual Nominee: 

Government staff - Maria Zimmer 

ASC 4th Quarter 2006 Individual Nominee: 

Government staff - Joan Henley 

The ASC/HP Quarterly Award honors individuals who 

have made significant contributions to the mission by 

exhibiting a high level of competence and efficiency in the 

performance of assigned duties and responsibilities while 

fostering teamwork. 

ASC/HP 3rd Quarter 2006 Individual Award 

Government staff - Maria Zimmer - For her leadership in 

developing the HPCMP Consolidated Customer Assistance 

Center (CCAC). 

ASC/HP 3rd Quarter 2006 Team Award 

Electronic Records Management (ERM) Team 

Government staff : Contractor staff:

 Bobbie Heaston Bridget Bussell

 Jacqulyne Person Duc Huynh

 Gretchen Prunty

 Jye Swan 

ASC/HP 4th Quarter 2006 Individual Awards 

Government staff:

Joan Henley – Comprehensive Security 

Assessment (CSA) Leadership 

Tim Sell – Supercomputing Conference 2006

 (SC’06) Leadership 

ASC/HP 4th Quarter 2006 Team Award 

Key Distribution Center (KDC) & ACE Server 

Consolidation Team: 

Government staff: Contractor Staff: 

Larry Falb Anita Christmas Dan Schornak

 Rich Gestrich Josh Dotson John Sprinkle

 Ron Trimble Delbert Hickman Jason Stare 

Vern Staats Jill Robertson Ann Ware

Pete Sampson Leslie Wilson

Outstanding Department of Defense (DoD) Employees 

with Disabilities Award honors outstanding Federal 

employees with disabilities and recognizes their 

contributions and achievements. 

Larry Falb – Individual Nominee 

CSC HPC COE Recognition “Dare to Soar” Award 

recognizes outstanding individuals, from each of the five 

CSC HPC COE organizations, who exemplify excellence in 

their support to researchers and customers throughout the 

nation.

Dr. Scott Kajihara – CSC Individual Nominee 

CSC Management 

Changes Announced 

Bill Asbury, Computer Sciences 

Corporation (CSC) Program 

Manager at the ASC MSRC since 

April 2006, has changed roles and 

now serves as a Senior Technologist 

for the CSC High Performance 

Computing Center of Excellence 

(HPC COE). Bill served at the ASC 

MSRC, in various capacities for 

more than ten years. 

Steve Baxter has assumed the role of Acting Program 

Manager for the ASC MSRC, after 

serving as the Deputy Program 

Manager since April 2006. Steve 

has served the ASC MSRC for ten 

years.

We wish Bill and Steve well in 

their new roles. All positions were 

effective April 9, 2007.

Spring 2007 Wright Cycles 15
Supercomputing 2006

Tampa, Florida 

16 Spring 2007 Wright Cycles 
Spring 2007 Wright Cycles 17
Hails and Farewells 

The ASC MSRC welcomes those who have recently filled 

staff positions. 

SeveralkeypersonnelhavelefttheASC 

Robyn

Estep,

CSC

Tim 

Hall,

Multimax

Jeremy

Goheen,

CSC

John

Hoffman,

ASC/HP

Steve

Hurley, 

CSC

Michelle

McDaniel,

CSC

Erin

O’Brien,

ASC/HPF

Kevin

Schoen,

ASC/HP

Tessa

Welton,

CSC

Barb

 McCauley, 

CSC

Tony 

West,

Multimax

Bill

Asbury,

CSC

MSRC recently. They will be missed. 

Eric

Graves,

ASC/HPF

Duc

Huynh,

Multimax

Dennis

Kahn,

ASC/HPF

Rick

Minamyer, 

ASC/HPF

Bill

O’Malley, 

ASC/HPF

Jason

Boudi,

ASC/HP

18 Spring 2007 Wright Cycles 
Spring 2007 Wright Cycles 19

Recent Visitors 

1.

2.

3. 4. 5.

6.

7. 8. 9.

10.

11. 

1. Mr. Doug Bowers, SES - AFRL/VA 

2. Dr. David Cronk - University of Tennessee 

3. Dr. David Hudak - Ohio Supercomputing Center (OSC) 

4. Dr. John Nehrbass - Ohio Supercomputing Center (OSC) 

5. Mr. Keith Seaman, SL - SAF/XCO, Washington DC 

6. Major General William Lord - SAF/XCI, Washington DC 

7. Dr. Gerald Noel - Central State University 

8. Dr. Steve Gorrell - AFRL/PRTF 

9. Mr. Robert Marcus - Central State University 

10. AVETEC visitors 

11. AFROTC - University of Pittsburgh 
PET Powering 

Tomorrow’s 

Solutions

MaXFlyer Near Space Vehicle

20 Spring 2007 20 Spring 2007 Wright Cycles 

The MaXFlyer is a concept for an unmanned, 

Near Space vehicle designed to meet 

today’s operational demands for dedicated 

communication, data, intelligence, 

surveillance, and recognizance (ISR) 

capabilities. It is a lighter than air vehicle 

intended to operate for a minimum 

of 6 weeks at 100k FT. It is to be 

transported by a CH47/C130 

to the battlefield and 

launched.

The initial 

aerodynamic analysis of 

the concept was performed 

using panel methods (low 

fidelity aero). However, the 

successful operation of the vehicle 

depends HIGHLY on the aerodynamic 

performance. Hence, more reliable and higher fidelity 

aerodynamic data was needed. A large HPCMP allocation 

was granted, and it was suggested that Cobalt would be the 

code to obtain this data. PET was contacted and developed 

and validated the meshes, results, etc. Extensive CFD 

simulations were used to size the propulsion system and 

other components. PET and MSRC personnel worked 

with Near Space Systems to design shapes that provide 

better aerodynamic performance, thus providing greater 

capability. The customer has traveled to WPAFB twice for 

collaborative work and training. 

This is a revolutionary system designed using simulation, 

and it is believed that there is major advancement in 

the control of the platform. Currently there are balloons 

that can reach that altitude and are viable for the time 

scales desired, but the problem is maintaining control 

and keeping the vehicle on station. Power and weight 

drive performance/capability, thus the system must 

be aerodynamically efficient so that minimal power is 

required to keep the vehicle on station. 

The use of CFD for development of the aerodynamic 

database facilitated the examination of a vast design space 

of aerodynamically efficient shapes, resulting in a vehicle 

system that meets size, shape, and propulsion requirements. 

This effort is very ‘lean’ in terms of time and cost, thus 

resources/capabilities were not available for a conventional 

design by test process. The ability to look at innovative, 

out of the box ideas was made possible by the simulation 

capability. 

For more information, please contact Near Space Systems, 

Inc. at 719-761-8835 or the ASC MSRC at 

asc.hp.outreach@wpafb.af.mil. 

BY DR. HUGH THORNBURG
A Consistent Computational Environment 

BY DR. DAVID CRONK, FAPOC, PET COMPUTATIONAL ENVIRONMENT TEAM

The Computational Environment (CE) Functional Area 

(FA) is a key component of the DoD HPCMP’s User 

Productivity Enhancement and Technology Transfer 

(PET) program. The Computational Environment 

encompasses all aspects of the user interface to DoD HPC 

resources, including but not limited to: programming tools 

(debuggers, libraries, solvers, higher order languages, 

performance analysis, prediction and optimization tools); 

distributed access mechanisms (queuing systems, clusters, 

distributed data and metacomputing); reusable parallel 

algorithms; and user access tools (portals and web-based 

access to HPC resources). 

A key goal of the CE functional area is to improve user 

productivity and application performance through provision 

of effective programming tools as part of a consistent, welldocumented computational environment. 

A Consistent Computational Environment (CCE): 

• Ensures that if a CCE software package is available 

on one HPCMP resource, it is available on all HPCMP 

resources

• Establishes installed software packages as the same 

version across resources 

• Installs software in the “same location” across 

resources

• Allows software, regardless of resources, to be linked/ 

used the same way 

• Makes it easy for users to get information about the 

installed tools 

The PET CE team deploys the Consistent Computational 

Environment on all allocated, unclassified machines across 

the four MSRCs (ARL, ASC, ERDC, and NAVO) as well 

as MHPCC and ARSC. 

CCE Deployment Matrix 

PET CE maintains a software repository with information 

about the type of software that is installed, on which 

resource, and how the software can be accessed. Users 

can view this repository by visiting http://rib.cs.utk.edu/

rib3app/catalog?rh=35. Information about each supported 

software package is available through this main repository 

page. Further, users can access a deployment matrix from 

this page. This deployment matrix (see Figure 1) shows 

in a single view which software packages are installed on 

which machines. Users can also use this deployment matrix 

to find information specific to a particular installation. For 

example, by clicking on “installed” at the intersection of 

“SuperLU” and “ERDC MSRC Cray XT3,” the user can 

find where the software is installed on that machine, as well 

as some usage information and the version number. 

ARL MSRC Intel 

EM64t Cluster 

ARL MSRC Intel 

Pentium Cluster 

ARSC IBM 

POWER4+ ASC MSRC HP XC ASC MSRC Origin 

3900 

ASC MSRC SGI 

Altix 

ERDC MSRC CRAY 

XT3 

ERDC MSRC SGI 

Origin 3900 

MHPCC Dell 

Cluster 

 Arpack Installed Installed Installed Installed Installed Installed Installed Installed Installed 

 ATLAS Installed Installed Installed Installed Installed Installed Installed Installed Installed 

 FFTW Installed Installed Installed Installed Installed Installed Installed Installed Installed 

GSL Installed Installed Installed Installed Installed Installed Installed Installed Installed 

HDF5 Installed Installed Installed Installed Installed Installed Architecture 

Not Supported Installed Installed 

 KOJAK Installed Installed Installed Installed Installed Installed Installed Installed Installed 

 LAPACK Installed Installed Installed Installed Installed Installed Installed Installed Installed 

MatlabMPI Installed Installed Installed Installed Installed Installed Installed Installed Installed 

netCDF Installed Installed Installed Installed Installed Installed Installed Installed Installed 

 Octave Installed Installed Installed Installed Installed Installed Installed Installed Installed 

Figure 1. CCE Software Package Deployment Matrix 

continued on next page 

Spring 2007 Wright Cycles 21
22 Spring 2007 Wright Cycles 

The deployment matrix is the best location to find the most 

up-to-date information about the availability of the CCEsupported tools on the supported resources. 

Software Location 

A CCE goal is to have all the CCE software installed 

in the same location regardless of center or machine. 

Unfortunately, different vendors have different default 

directory structures, rendering it impractical to have the 

CCE software installed under the same physical directory 

on each machine. This limitation is addressed by making 

the CCE directory “look” the same on each machine. 

This is accomplished by having each center define an 

environment variable, $PET_HOME, on each machine. 

On login, this environment variable is automatically set to 

point to the CCE installation directory. 

Standard sub-directories for libraries, executables, and 

header files reside under $PET_HOME. These directories 

include, bin32, bin64, lib32, and lib64. Additionally, bin 

and lib are linked to the default directories (bin32 and 

lib32 on 32-bit architectures and bin64 and lib64 on 64-bit 

architectures).

This standard structure allows users to omit fully qualified 

paths from scripts. As an example, to find the header files 

for a needed software package, users need to simply add 

–I($PET_HOME/include) to their makefiles. 

Similarly, to link, users simply need to add –L($PET_ 

HOME/lib) (or lib32 or lib64) to their makefiles and 

the normal –l<lib_name> to their flags. To access CCE 

executables, they can simply add $PET_HOME/bin (or 

bin32 or bin64) to their paths. 

While every effort has been made to use this structure for 

all the CCE packages, some packages simply do not lend 

themselves to this structure. MatlabMPI and PerfStats can 

both be found in their specific directory structure under 

$PET_HOME/pkgs. Similarly, TAU can be found in its 

specific directory structure by accessing $PET_HOME/tau. 

Consistent Software Suite 

As part of the CCE, the Computational Environment team 

has identified a suite of tools and libraries to be deployed 

across the six HPCMP allocated centers. This software suite 

includes parallel programming tools, libraries, and problem 

solving environments: 

Performance Analysis Tools: 

• PAPI – provides the tool designer and application 

engineer with a consistent interface and methodology 

for use of the hardware performance counters found in 

most microprocessors. PAPI enables software engineers 

to see the relationship between software performance and 

processor events. 

• PapiEx – serves as a performance analysis tool 

designed to transparently measure the hardware 

performance counter values for an application. It uses 

library pre-loading to intercept process/thread creation. 

PapiEx measures the entire run of an application. By 

default, this includes all sub-processes. It can optionally 

monitor threads and do counter multiplexing, and it has a 

host of other features. 

• PerfStats – includes a suite of tools for doing statistical 

analysis of performance data. These tools are designed 

to assist the user in identifying potential performance 

problems.

• TAU – gathers performance information through 

Figure 2. Graphical display of performance analysis results from TAU’s profile visualization tool 
Spring 2007 Wright Cycles 23

instrumentation (either hand or automatic) of functions, 

methods, loops, basic blocks, and statements. The API 

provides selection of profiling groups for organizing and 

controlling instrumentation. TAU’s profile visualization 

tool provides graphical displays of all the performance 

analysis results (see Figure 2). 

• KOJAK – Supports automated tracefile analysis for 

applications written in MPI, OpenMP, SHMEM, and 

combinations thereof. KOJAK uses a pattern-matching 

approach to automatically pinpoint performance 

bottlenecks of various types. 

Scientific Data Access 

• HDF5 – serves as a general purpose library and 

file format for storing scientific data. HDF5 can store 

two primary objects: datasets and groups. A dataset is 

essentially a multidimensional array of data elements, and 

a group is a structure for organizing objects in an HDF5 

file. Using these two basic objects, one can create and 

store almost any kind of scientific data structure, such as 

images, arrays of vectors, and structured and unstructured 

grids. Users can also mix and match them in HDF5 files 

according to their needs. 

• Network Common Data Form (netCDF) – provides an 

interface for array-oriented data access and a library that 

allows an implementation of the interface. The netCDF 

library also defines a machine-independent format for 

representing scientific data. Together, the interface, library, 

and format support the creation, access, and sharing of 

scientific data. 

Problem Solving Environments 

• GNU Octave – serves as a high-level language, 

primarily intended for numerical computations. It provides 

a convenient command line interface for solving linear 

and nonlinear problems numerically, and it performs other 

numerical experiments using a language that is mostly 

compatible with MATLAB. It may also be used as a batchoriented language. 

• MatlabMPI – a set of MATLAB scripts that implements 

a subset of MPI and allows any MATLAB program to 

be run on a parallel computer. The key innovation of 

MatlabMPI is that it implements the widely used MPI “look 

and feel” on top of standard MATLAB file I/O, resulting 

in a “pure” MATLAB implementation that is exceedingly 

small (~300 lines of code). Thus, MatlabMPI will run on 

any combination of computers that MATLAB supports. 

Debuggers

• Valgrind – provides a suite of tools for debugging and 

profiling on Linux systems. With the tools that come with 

Valgrind, users can automatically detect many memory 

management and threading bugs, avoiding hours of 

frustrating bug-hunting, which stabilizes the programs. 

Users can also perform detailed profiling to speed up and 

reduce memory usage. 

Open Source Math Libraries 

A set of open source math libraries has also been identified 

and included within the scope of CCE. This suite of open 

source math libraries includes: 

• ARPACK 

• FFTW 

• The GNU Scientific Library (GSL) 

• PETSc 

• SuperLU 

• ScaLAPACK 

• SPRNG 

User Assistance 

User questions about how a CCE software package 

is installed, (such as an experienced user noticing a 

problem with the software or requirements for a unique 

configuration), should be directed to the Help Desk for the 

center at which they are trying to use the software or the 

Consolidated Customer Assistance Center (CCAC) once it 

is available. The PET CE team will then work with both the 

user and customer support to address the problem. 

Questions related to the use of any CCE software tool or 

library for a particular application should be directed to the 

appropriate PET CTA team. This contact information can 

be found by visiting the PET Online Knowledge Center 

(OKC) at https://okc.erdc.hpc.mil/index.jsp. The PET CE 

team will then work with the CTA-based team as well as 

the user to ensure adequate support. 

For other inquiries, including inputs for recommendations 

for new tools to be added to the CCE Suite, requests for 

training, or advanced assistance using a tool or library, the 

PET CE team can be contacted directly by sending e-mail 

to: pet-ce@cs.utk.edu. Every effort will be made to respond 

to requests for assistance within 24 hours. 
24 Spring 2007 Wright Cycles 

SIP High Productivity Tool 

Signal Image Processing (SIP) users often need to run 

many jobs involving different angles. For example, 

Xpatch users may desire to run many Xpatch jobs 

over a variety of elevation and azimuth angles. While 

each job is independent of the other and can be run in 

parallel, managing them in a batch oriented setting can 

be painstaking. Running Xpatch over a half sphere, for 

example, with one degree increments could involve 

360 x 90 = 32,400 jobs. 

Tracking such a collection of jobs can be challenging in 

a batch oriented environment. For example, it is possible 

that some jobs may abort due to CPU time limits. Tracking 

what has completed successfully can be daunting. The 

SIP High Productivity Tool (SIPHPT) helps folks submit 

jobs to be run in parallel and tracks jobs that have already 

completed. In the case of aborted jobs, the SIPHPT can be 

instantiated as needed until all jobs have completed. The 

tool will only run those jobs that have not been run so that 

the user need not worry about tracking individual jobs. 

Also, the nomenclature SIPHPT uses for filenames and 

subdirectories eases the task of determining the current 

status of one’s job. 

Although the current version of the tool is oriented toward 

Xpatch, it can be used with other programs as well. 

Interested users can contact PET SIP On-Site Dr. Bracy 

Elton (elton@osc.edu, bracy.elton@wpafb.af.mil, 

937/904-5139, DSN 674-5139) for assistance in using this 

tool.

BY DR. BRACY ELTON 

Visualization of look 

down angles on test 

vehicle

Former PET Intern Wins Award 

Philip Wheat was awarded a National Science Foundation 

Graduate Research Fellowship, one of the most prestigious 

awards for students seeking advanced degrees in science and 

engineering. He is currently a graduate student at Arizona State 

University, pursuing a Ph.D. in mechanical engineering. Philip 

spent two summers in his undergraduate years working as a 

PET intern at the ASC MSRC. The first summer he focused 

on validation and verification of the Air Force Research 

Lab’s (AFRL) computational electromagnetic code, and the 

following summer he presented his research results at the 

HPCMP’s Users Group Conference. 

ASC MSRC 

Congratulates

Philip Wheat 
Upcoming

Conferences

June 18 - 22, 2007 

DoD Users Group Conference 

Omni William Penn 

Pittsburgh, Pennsylvania 

www.hpcmo.hpc.mil/UGC2007/index.html 

July 23 - 25, 2007 

Commodity Cluster Symposium 2007 (CCS2007) 

Sheraton Annapolis Hotel 

Annapolis, Maryland 

www.clustersymposium.org 

August 5 - 9, 2007 

SIGGRAPH2007

San Diego Convention Center 

San Diego, California 

www.siggraph.org/s2007 

October 27 - November 1, 2007 

IEEE Visualization 2007 

Hyatt Regency Sacramento 

Sacramento, California 

http://vis.computer.org/vis2007 

November 10 - 16, 2007 

Supercomputing Conference 2007 

Reno Sparks Convention Center 

Reno, Nevada 

www.sc07.supercomputing.org 

November 26 - 30, 2007 

2007 Materials Research Society (MRS) Fall Meeting 

Hynes Convention Center and 

Sheraton Boston Hotel 

Boston, Massachusetts 

http://lucy.mrs.org/

Wright Cycles 

Spring 2007

Chief Editor 

Rachel Crouse

Layout

Justin Sanker

Graphics

Chuck Abruzzino

Editorial Staff 

Pat Dettling

Jeff Graham

Joan Henley

Dinah Luneke

Steven Wourms

Maria Zimmer

The Aeronautical Systems Center (ASC)

Major Shared Resource Center (MSRC) is

a computational science facility supporting

Department of Defense (DoD) research,

development, and test and evaluation

communities with high performance

computing resources. Created as part of

the DoD’s High Performance Computing

Modernization Program (HPCMP), the

ASC MSRC High Performance Computing

Center is located on Wright-Patterson Air

Force Base (WPAFB) and is one of four

DoD MSRC sites. Computer Sciences

Corporation (CSC) is the prime support

contractor at the ASC MSRC.

Wright Cycles is published by the CSC

Customer Assistance and Technology

Center. Your comments, ideas and

contributions are welcome. Any opinions,

findings, conclusions or recommendations

expressed in this publication are those of

the author(s) and do not necessarily reflect

the views of the DoD. All photographs were

taken byASCMSRCstaff,unless otherwise

noted.

ASC MSRC

www.asc.hpc.mil

asc.hp.outreach@wpafb.af.mil

Approved for Public Release

Distribution Unlimited

ASC 07-0178
Wright Cycles 

ASC/HP Bldg 676

2435 Fifth Street

WPAFB OH 45433-7802 

PRSRT STD

US POSTAGE

PAID

DAYTON OH

PERMIT NO. 1161