# Presentation: 035535

## Los Alamos Cluster Visualization

**Allen M****c****Pherson**
**Los Alamos National Laboratory**
**August 13, 2001**

## Agenda

**Volume rendering overview**
**Cluster-based volume rendering algorithm**
**Back-of-the-envelope analysis**
**Cluster architecture**
**Software environment**
**Recent results**
**Future work**

## What is Volumetric Data?

**3-D grid or mesh**
**Data sampled on grid**
**Samples called “voxels”**
**Many grid topologies**
- Structured
- E.g. rectilinear
- Unstructured

## How is Volume Data Generated?

**Sensors**
- CT scanners
- MRI
**Simulations**
- Fluid dynamics
**Measured data**
-  Ocean buoys

## Looking at Volumetric Data

**Constant value surface**
- Isosurface algorithm
- Polygonal data generated
- Don’t see entire volume
- Polygons usually generated in software
- Polygons rendered with hardware

## Looking at Volumetric Data

**True volume rendering**
- Treat field as semi-transparent medium
- “blob of Jello”
- Can see entire volume

## Transfer Functions

**Indirectly maps data to color and opacity**
**Allows user to interactively explore volume**

## Software Volume Rendering

**Ray casting**
- Image order algorithm
- Trace ray through image plane and into volume
- Sample volume at regular intervals along ray
- Combined samples yield ray’s pixel value (compositing)

## Hardware Volume Rendering

**Software approaches are too slow**
- Interactivity required for exploration
**Use texture mapping hardware to accelerate**
- Textures emulate the volumetric data
- Hardware lookup tables accelerate transfer function updates
**Use parallelism for large volumes (multiple hardware pipes)**

## Texture Mapping Approach

**Texture is volume**
- 3-D texture
- Many 2-D textures
**“****Cleave” 3-D volume with slice planes**
**Composite resultant images in order**
**Essentially parallel ray casting**

## Early Experience at Los Alamos

**Problem: visualize large volumetric data (1024****3****) interactively**
**Use texture-based approach for speed**
**Single pipe can’t handle large volumes**
**Use multiple pipes in combination to render large volumes**

## Large SGI-based Solution

**128 processor Onyx 2000**
**16 Infinite Reality graphics pipes**
**1 Gvoxel volume rendered at 5 Hz**
**Want to accomplish the same goal (or better) using less expensive, commodity-based, solution**
**Our volumes will get bigger—8K****3****!**

## Cluster-based Solution

**Algorithm similar to large SGI solution**
- Break volume into smaller sub-volumes
- Use many PC nodes with commodity graphics cards to render sub-volumes
- Read resultant images back and composite in software using interconnected cluster nodes
- Organize as pipeline for speed

## Algorithm Schematic

- UI

- R

- R

- R

- R

- C

- C

- C

- C

- UI

- Transformation

- Rendered
- Sub-Image

- Compositing
- Traffic

- Composited
- Sub-Images

## Serial vs. Pipelined

- Serial

- Pipeline

- UI

- R

- C

- Frame 1

- Frame 0

- Frame 2

- Frame Time

- C

- R

- UI

- C

- R

- UI

- C

- R

- UI

- Frame 0

- Frame 1

- Frame 2

- Frame
- Time

- Latency

## Pipeline Issues

**Frame time = time of longest stage**
- Need to _**balance**_ stage times
**Deep pipelines can induce long latency**
- Keep pipelines short
**“****Circularity” of pipeline is troublesome**
- Communications programming is tricky

## Back-of-the-Envelope

**Analyze feasibility**
- Examine “speeds and feeds” of each component
- Test against theoretical numbers wherever possible
- Won’t guarantee success, but gets us in the ballpark

## Cluster Components

**Initial hardware selections**
- CPU: dual Intel
- want commodity PC
- Graphics: Intense 4210
- using 3-D texture
- Network: GIG-E
- Fast commodity network
- Reusable at completion of project

## Bounding Parameters

**Graphics card texture memory**
- Dictates size of volume that can be rendered
**Graphics card fill rate**
- Dictates speed of actual volume rendering
**Framebuffer readback rate**
- How fast rendered sub-frame can be read to host
**Network speed**
- How fast images can be moved through the cluster

## Bounding Parameters (theory)

**Node**

**CPU (2)**

**Memory**

**Graphics Card**
**240 Mpix/sec fill**

**Texture Mem**
**128 MB**

**AGP-2**
**512 MB/sec**

**GIG-E 125 MB/sec**

## Bounding Parameters (tested)

**Node**

**CPU (2)**

**Memory**

**Graphics Card**
**240 Mpix/sec fill**

**Texture Mem**
**128 MB**

**AGP-2**
**280 MB/sec**

**GIG-E 55 MB/sec (MPI)**

## Data Magnitude

## Limit 1: Rendering

**240 Mtex/sec**
- At 5 FPS budget ~50 Mtex/frame
- 1-1 pixel-voxel gives 50 Mvoxel volume
- ~512x512x256 (64 MB through TLUT)
- 32 nodes gives 2 Gvoxel volume
- Theoretical number
- Conservatively use 1⁄2 of theoretical
- Back to 1 Gvoxel volume

## Limit 2: Image Readback

**280 MB/sec AGP-2 tested**
- Assume that we render into a 10242 image
- Matches volume resolution to screen resolution
- RGBA gives 4 MB/frame
- 280/4 = 70 FPS
- Well within budget

## Limit 3: Network Performance

**55 MB/sec tested on GIG-E with MPI**
- 4 MB (or smaller) images
- 55/4 = 11 FPS
- Within budget, but...
- May need to transport image multiple times per frame (render, composite, display)
- 5 FPS allows only two image moves—may not be fast enough

## Limit 4: Volume Download

**Only required for time-variant data**
- 64 MB volume from Limit 1
- At 5 FPS requires 320 MB/sec download
- Tested AGP-2 limits to 280 MB/sec
- Would need matching I/O
- 320 x 32 nodes:  10 GB/sec aggregate I/O

## Balanced Pipeline Stages?

**UI**
- Very fast, small data transfers (transform, TLUT)
**Render**
- 200 ms/frame + 4 MB image transfer
**Composite**
- Composite operations + 4 MB image transfer
**Pipeline forces equal stage lengths**
**Network time needs to be considered**

## Los Alamos KoolAid Cluster

## Cluster Compute Hardware

**36 Compaq 750**
- Shared rendering/compositing nodes
- 4 nodes used for UI and development
- Dual 800 MHz Xeon
- 1 GB RDRAM per node
- Intel Pro-1000 GIG-E card

## Cluster Compute Issues

**Intel 840 chipset allows simultaneous:**
- AGP transfers
- Network transfers
- CPU/memory interaction
**Some problems with chipset**
- Poor PCI performance when compared to Serverworks—slows networking

## Cluster Network Hardware

**Extreme GIG-E switch**
- Supports jumbo packets
- Full speed backplane
- Simultaneous point-to-point transfers
- Intel Pro-1000 GIG-E cards
- Tested for this application

## Cluster Network Issues

**GIG-E is **_**relatively**_** slow and inefficient**
- Protocol processing eats CPU
- Extreme switch is _expensive_, but nice
**Need to test actual communications patterns**
- Simple “netperf” style is not enough
- Test with communications library to be used (MPI)
**Numerous driver issues—test, test, test!**
**All GIG-E equipment is re-usable**

## Cluster Graphics Hardware

**3Dlabs Wildcat 4210**
- 128 MB texture memory
- 128 MB framebuffer memory
- 3-D texture hardware

## Cluster Graphics Issues

**Sub-optimal compared to recent alternatives**
- Poor fill rate
- AGP-2 interface
- Expensive: ~$4000/card
- Lacks nifty new features (DX8, etc.)
**Can clearly do better next time**

## Software Environment (OS)

**Windows 2000**
- Not a religious issue with us
- Only OS with driver support for Wildcat 4210
- Best bet for drivers (commodity cards)
- Most application code portable to Linux
- Can experiment with DX8 features later

## Software Environment (Rendering)

**OpenGL**
- 3-D textures for volume rendering
- Not in pre-DX8 versions from Microsoft
- Solid support on Wildcat 4210
**Software compositing**
- Have CPUs with nothing to do
- Completely general for future experimentation

## Software Environment (Networking)

**MPI**
- Argonne MPICH implementation
- Easy to learn and use
- Implementation adds opaque layer which makes troubleshooting difficult
- A few Win2K issues
- General lack of tools (e.g. log viewing)
- Tag limit of 99 (MS licensing??)

## Results

**To be presented at Siggraph 2001**
- See _[www.acl.lanl.gov/viz/cluster](http://www.acl.lanl.gov/viz/cluster)_ for latest

## Future Work

**Clusters of task-specific mini-clusters**
- Rendering, compositing, I/O, display
- Possibly specialized interconnect between clusters
- DVI
- Fiber Channel
- Optimal interconnect for individual mini-clusters
- Myrinet-2000
- Simple 100 Mb Ethernet

## Future Work (Rendering Cluster)

**Take rendering cluster to 64 nodes**
- Still Compaq 750s
- New nVidia/ATI cards when 3-D texture-capable
- May use Microsoft DirectX 8 vs. OpenGL
- Doesn’t need high speed interconnect
- Just transforms and TLUTs
- _Does_ need high speed connection to compositing cluster

## Future Work (Compositing Cluster)

**64 1U compositing nodes**
- Dell PowerEdge 1550
- Single 1 GHz PIII
- Serverworks chipset
- Interconnected with Myrinet-2000
- 2 Gb/sec interconnect
- Much faster than GIG-E, much less CPU overhead
- May run Linux
- No need for Win2K since no graphics cards

## Acknowledgements

**John Patchett**
**Pat McCormick**
**Jim Ahrens**
**Richard Strelitz**
**Joe Kniss (University of Utah)**