# Presentation: 720660

## Future Directions in Computer and Systems Architecture for Scientific Computing

- Rick Stevens
-  Director, Mathematics and Computer Science Division
- Argonne National Laboratory
- and
- Professor of Computer Science and
- Co-Director of The Computation Institute
- The University of Chicago
- stevens@mcs.anl.gov

## Outline of This Talk

- Supercomputing trends
- Possible Paths to PetaFLOPS
- Where are Clusters Going
- What’s after PC Clusters
- What’s this thing called the Grid
- Questions

## The National Academic Usage of Supercomputers Continues To Grow Exponentially

**Source: Quantum Research; Lex Lane, NCSA**

- 1000 x 1985

- Teraflops

- Gigaflops

## Basic

- Algorithms
- &
- Numerical
- Methods

- PDE

- Transport

- ODE

- Fields

- Symbolic
- Processing

- Pattern
- Matching

- Raster
- Graphics

- Monte
- Carlo

- Discrete
- Events

- n-body

- Fourier
- Methods

- Graph
- Theoretic

- CFD

- Reaction-Diffusion

- Radiation

- Combustion

- Structural Mechanics

- Multibody
- Dynamics

- Electromagnetics

- Geophysical
- Fluids

- Weather and Climate

- Aerodynamics

- Reservoir
- Modelling

- Flow in
- Porous Media

- Ecosystems

- Multiphase Flow

- CVD

- Plasma
- Processing

- Astrophysics

- Seismic
- Processing

- Cloud Physics

- Chemical
- Reactors

- Boilers

- Chemical
- Reactors

- Magnet Design

- Economics
- Models

- Phylogenetic Trees

- Electrical Grids

- Pipeline Flows

- Distribution Networks

- Biosphere/Geosphere

- Neural Networks

- Crystallography

- Tomographic
- Reconstruction

- MRI Imaging

- Diffraction
- Inversion
- Problems

- Signal
- Processing

- Molecular
- Modeling

- Condensed Matter
- Electronic Structure

- Quantum
- Chemistry

- Electronic
-  Structure

- Rational
- Drug Design

- Biomolecular
- Dynamics

- Nanotechnology

- Data
- Assimilation

- Chemical
- Dynamics

- Atomic
- Scattering

- Actinide
- Chemistry

- Fracture
- Mechanics

- Cosmology

- Astrophysics

- Orbital
- Mechanics

- Military
- Logistics

- Manufacturing
- Systems

- Population
- Genetics

- Air Traffic
- Control

- Transportation
- Systems

- Economics

- VLSI
- Design

- QCD

- Nuclear Structure

- Neutron
- Transport

- Virtual
- Reality

- Virtual
- Prototypes

- Computational
- Steering

- Scientific
- Visualization

- Multimedia
- Collaboration
- Tools

- Genome
- Processing

- Computer
- Vision

- Databases

- Data Mining

- Cryptography

- Intelligent
- Search

- Computer
- Algebra

- Number Theory

- Automated
- Deduction

- Intelligent
- Agents

- CAD

## Can PetaFLOPS Transition us from Computing as a means for Explanation to Computing as a mode for Discovery?

- Two Modes of Discovery
- getting to a new place first!!
- staying in the new place with the time to look around
- Needed Environment
- reliable infrastructure (discovery happens at the edges)
- predictable and abundant availability of resource (time)
- freedom from pressure to avoid failure (dumb ideas)
- To make this transition we need to make scientific computing resources more abundant, more available and more usable

## Best Prepared Applications Communities

- Astrophysics
- Stars, Supernovae, Neutron Stars, Cosmology
- Weapons Simulations
- Burn Codes, 3D Hydrodynamics, 3D Structures
- Climate and Weather Modeling
- Computational Chemistry
- Molecular Modeling
- Cryptography

## Less Prepared Applications Communities

- Computational Linguistics
- Computational Economics
- Operations Research
- Bioinformatics
- Computational Logic
- Complex Systems Simulation
- Engineering/Design Simulations

## Slide 8

## SIDE VIEW

- 4oK
- 50 W

- 77oK

- 1.4 m

- 0.3 m

- Fiber/Wire Interconnects

- 1 m

- 3 m

- 0.5 m

- 1 m

## SIDE VIEW

- 4oK
- 50 W

- 77oK

- Fiber/Wire Interconnects

- 3 m

- 0.5 m

- 220Volts

- Nitrogen

- Helium

**Hard Disk**
**Array**
**(40 cabinets)**

- 3 m

**Tape Silo**
**Array**
**(400 Silos)**

**Front End Computer Server                            **

- Console

- Cable Tray Assembly

**Generator**

**WDM Source**

**Optical Amplifiers**

- 220Volts

**980 nm Pumps**
**(20 cabinets)**

**Generator**

## “Simplified” Application View of an HTMT “Slice”

- DPIM

- VORTEX

- SPIM

- RTI

- SPELL
- CRAM

- CNET

- SPELL
- CRAM

- bw = 8 GB/s/PIM CPUlogical
- latency =  85 ns + software

- bw =  512GB/s/CPUlogical
- latency = 20+ ns

- bw = 64 GB/s/CPUlogical
- latency = 20 ns

- 512K CPUlogical units, 128 per slice
- 500 MHz clock
- 32 MB/CPUlogical

- 256K CPUlogical, 64 per slice
- 1 GHz clock
- 4 MB/CPUlogical

- 4096 units, 1 per slice
- 256 GHz clock
- 1 MB CRAM/CPUlogical

- 10 GB/s  DPIM  DV

- 128 DPIM CPUs
- 4 GB RAM

- 64 SPIM CPUs
- 256 MB RAM

- 40 GB/s  DV  SPIM

- 512 GB/s   SPIM  SPELL

- 1 SPELL
- 1 MB CRAM

## IBM’s Blue Gene Machine

- 1 Million CPUs
- 1 Gflops/CPU, 64 bits, add/mult/cycle @ 500 MHz
- 32 CPUs per chip, 16 MB memory on Chip (32 banks), 5 cycle access
- 57 RISC instructions, 8 register banks, 8 way threaded (8 M threads)
- 6 I/O Channels per die, 8 bits wide, 2 GB/s bi-directional
- DMA engine per channel, active messaging hardware support
- external memory interface per chip (not used on the internal nodes)
- 3D Mesh topology with dynamic routing
- self healing capability (routes around back CPUs and Chips)
- edge chips will have external memory and PC interfaces
- 64 chips per board  2K CPUs per board  32K CPUs rack
- 32 Racks  Air cooled, about 2,000 sq. ft floor space
- Chips in design phase today.. Chip Production starts in ‘01

## IBM Blue Gene .. Software and Applications

- Compiler (C and C++) based on PowerPC Compilers
- Communications Library (simple message passing)
- Dynamic Source Routing, Active Message Like HW support
- Simple Node Exec (each node only has 500K memory)
- Application Target  Protein Folding.. Brute Force
- One Chip per Atom (~32K atoms)
- Bonded Force Terms Computed Mostly Locally
- Non-bonded Terms computing via messaging
- Other possible applications targets
- N-body problems in cosmology?
- Emergent Phenomena (network sim, Alife, Automata Theory)

## Chiba City

**8 Computing Towns**
- 256 Pentium III systems

**1 Visualization Town**
-   32 Pentium III systems
-  with Matrox G400 cards

**1 Storage Town**
-   8 Xeon systems
- with 300G disk each

**Cluster Management**
-   12 PIII Mayor Systems
- 4 PIII Front End Systems
-  2 Xeon File Servers
- 3.4 TB disk

**High Performance Net**
-   64-bit Myrinet

**Management Net**
- Gigabit and Fast Ethernet
- Gigabit External Link

- The Argonne Scalable Cluster

## Chiba City System Details

- Purpose:
- Scalable CS research
- Prototype application support
- System - 314 computers:
- 256 computing nodes, PIII 550MHz, 512M, 9G local disk
- 32 visualization nodes, PIII 550MHz, 512M, Matrox G200
- 8 storage nodes, 500 MHz Xeon, 512M, 300GB disk: 2.4TB total
- 10 town mayors, 1 city mayor, other management systems: PIII 550 MHz, 512M, 3TB disk

- Communications:
- 64-bit Myrinet computing net
- Switched fast/gigabit ethernet management net
- Serial control network
- Software Environment:
- Linux (based on RH 6.0), plus “install your own” OS support
- Compilers: GNU g++, etc
- Libraries and Tools: PETSc, MPICH, Globus, ROMIO, SUMMA3d, Jumpshot, Visualization, PVFS, HPSS, ADSM, PBS + Maui Scheduler

## Slide 16

## Example: 320-host Clos topology of 16-port switches

- 64 hosts

- 64 hosts

- 64 hosts

- 64 hosts

- 64 hosts

- (From Myricom)

## Evolution of Cluster Services

**Login**
**File Service**
**Scheduling**
**Management**

- 1 computer that is distinct from the computing nodes.

- 1 or more computers per service.

- Improve computing performance.

- Improve system reliability and manageability.

**File Service**
**Scheduling**
**Management**

**Login**

**...**

**Login**

**File Service**

**Login**

**File Service**

**Scheduling**

**Management**

**...**

**The cluster grows...**

**Server**
**Strategy:**

**Basic**
**Goal:**

**Usage &**
**Management**
**Model**

**Scheduling**
**Management**

## Alarming Trends of Systems Software Availabilityfrom Computing Hardware Vendors

**Applications**

**High Perf Tools**

**High Perf Env**

**System Software**

**Operating System**

**Hardware**

- Solution
- provided by:

**Applications**

**High Perf Tools**

**High Perf Env**

**System Software**

**Operating System**

**Hardware**

**Customer**

**Vendor**

**1990**

**2000**

***2010***

- Solutions
- Created by...

- Focusing on
- Scientific
- Computing

- Leveraging
- Business
- Solutions

- Some
- Government
- Research

- GigaFLOPs

- TeraFLOPs

- PetaFLOPs

- Necessary
- computing
- power:

**Notes:**
Talking points for this slide:
  The ASCI barriers story
  The reasons that vendors have decreased support and their realities
  The need for Petaflops

## HPC Community is Leveraging Commodity Hardware 

- But has not yet been able to significantly leverage commodity software or even commodity software ABI/APIs beyond the basic kernel and OS services
- commodity software components don’t yet exist to build scalable systems software
- high performance is not the primary goal of the few commercial efforts building systems software type infrastructure for commodity environments
- High-end community has a perennial labor shortage for HPC systems software developers
- How to establish a critical mass of systems software developers focused on the high-end?

## Targeted Hardware Systems

- End User Integrated Clusters (10K-100K nodes)
- low-cost high-volume CPUs and motherboards (SMP 8-16)
- low-cost high-volume communications and SANs
- relatively simple memory subsystems and I/O support
- Vendor Integrated Systems (10K nodes)
- high-performance CPUs and packaging (SMP > 16)
- proprietary SANs and communications and interfaces
- sophisticated memory and I/O subsystems
- New Generation Architectures (1K-1M nodes)
- unique CPUs, ISAs and packaging
- unique SANs and communications interfaces
- ultra-sophisticated memory and I/O subsystems

## Possible Software Stack for HPC

## Node Issues

- Floating Point and Integer Performance
- Memory Bandwidth
- Cache Performance and Cache Sizes
- Motherboard Chip Sets

## Some Cluster Node Options Today

- Intel Pentium III
- AMD Athlon
- Compaq Alpha
- Intel IA-64
- IBM PowerPC Power4

## Intel Pentium III

- Now available at 1.0GHz (1000MHz) for the desktop
- 866, 850, 800, 750, 733, 700, 667, 650, 600, 550, 533, 500 and 450 MHz Clock Speeds
- 70 New Instructions (since first Pentium)
- P6 Architecture
- 133- or 100-MHz System Memory Bus
- 512K Level Two Cache or 256K Advanced Transfer Cache
- Intel's 1GHz Pentium III is shipping now at a list price of $990.
- For more information, check out Intel's Web site at www.intel.com/PentiumIII/.

## IA-32 Willamette Demo’d @ 1.5 GHz

- Willamette bus is a source-synchronous 64-bit 100MHz bus that is quad-pumped delivering a total of 3.2GB/s of bandwidth--three times the bandwidth of the fastest Pentium III bus.
- A unique and unexpected aspect of Willamette's microarchitecture is its "double-pumped" ALUs. Claiming the effective performance of four ALUs, the two physical ALUs are each capable of executing an operation in every half-clock cycle.
- 20 Stage Pipeline compared to 10 on P6;
- SSE 2 includes support for (dual) double-precision SIMD floating-point operations and uses the MMX register set.

## AMD Athlon v Pentium III

- Pentium III's smaller, but faster, on-chip L2 cache is superior to Athlon's larger, but slower, external L2 cache for most benchmarks.
- Pentium III's cache scales with core CPU frequency, while Athlon's does not. Athlon's deficiency in this regard will not be remedied until AMD delivers Thunderbird, which will have a 256K on-chip L2.

- (Source: AMD, Intel,  MDR)

## AMD K7/Athlon (www.amd.com)

- Up to 72 instructions can be in execution in K7's out-of-order integer pipe (light purple), floating-point pipe (dark purple), and load/store unit. (Source AMD and MDR)

- K7's 22 million transistors occupy 184 mm2 in a slightly enhanced version of AMD's 0.25-micron 6-layer-metal CS44E process. (Source: AMD)

- AMD is shipping 1GHz Athlon processors. List price for the 1GHz version is ~$1,000 in quantities of 1,000 units.

## Compaq Alpha 21264/21364

- The 733-MHz four-issue out-of-order 21264 is today’s fastest microprocessor
- The 21364, code-named EV7, will use multiple Rambus channels to greatly reducing memory latency.
- The 21364 will use the 21264 core but increase frequency to 1 GHz with a 0.18-micron process, add a 1.5M on-chip L2, a 6-GByte/s memory port, and 13 GBytes/s of chip-to-chip bandwidth.

- (Source Compaq, MDR)

- (Source Compaq, MDR)

## Intel IA-64

- NEW Intel/HP 64-bit microprocessor
-  New Architecture Influenced by PA-RISC
- RISC, VLIW, SuperPipelined, Superscalar
- Executes both IA-32 and IA-64 code
- 3 Instructions (41 bit) per bundle
- moderate VLIW, 128 bits/bundle
- Multiple Instruction bundle issues/clock (20 instructions in flight)
- Pipeline depth of 10
- Advanced and Speculative Mechanisms
- Memory Hierarchy Hints
- Deep Branch Prediction

## Intel IA-64 Building Blocks

## Future IBM PowerPC (Power4)

- 170-million-transistors
- Two 64-bit 1-GHz five-issue superscalar cores
- Triple-level cache hierarchy
- A 10-GByte/s main-memory interface
- 45-GByte/s multiprocessor interface.
- IBM will see first silicon on Power4 in 1Q00, and systems will begin shipping in 2H01.

## IBM’s Power4

- Power4 includes two >1-GHz superscalar cores
- > 100 GBytes/s of bandwidth to a large shared-L2 cache
- > 55 GBytes/s of bandwidth to memory and other Power4 chips
- Four Power4 chips packaged in a MCM as an eight-processor SMP with total bandwidths of 40 GBytes/s to memory and 40 GBytes/s to other modules.

- (Source IBM, MDR)

- (Source IBM, MDR)

## Conclusions

- What is the best Node for your cluster?
- Well it depends on your application of course
- For Integer codes both Pentium III and Athlon @ 1 GHz+
- For 32 bit floating point IA-32 probably best price performance today
- For 64 bit floating point Compaq Alpha is the clear lead
- How long will these choices be clear ?
- Probably Not very long
- Intel will introduce IA-64 (Itanium) very shortly (mid-summer)
- ~3 GFLOPS @ 800 MHz for 64 Bit, ~6 GFLOPS for 32 bit
- Limited by Front Side Bus bandwidth of ~ 2.1 GB/s
- AMD’s Thunderbird will address external cache issues with Athlon
- AMD is targeting Sledgehammer at IA-64
- Compaq is revising Alpha (EV67-> EV7) to compete with IA-64
- IBM is putting considerable resources into Power4, and CMP and MCM’s will enable IBM to produce very powerful servers/cluster nodes

## Possible Extreme Linux Systems in 2010 (conservative)

- Possible Extreme Linux Systems in 2010 (conservative)

- 4.8 GHz Clocks
- 16 way SMP node
- 300 Gflops node
- > 4 GB RAM
- >160 GB disk
- 8 x 400+ MB/s interfaces/node
- < 1 kW power/ node
-  $2000/node + $2000 other
- 8000 nodes
- 2.5 PetaFLOPS peak
- Total system Price ~$16M
- Annual power bill ~$8M

## Some Cluster Node Options Tomorrow

## Compaq Itsy Pocket Computer

- Hardware
- 200 MHz StrongARM SA-1100 Processor
- 16MB DRAM, 4MB Flash Memory
- Audio CODEC, microphone, speaker
- LCD Display and Touch screen
- Serial, IrDA and USB IO, pushbuttons
- Software
- Runs Linux
- MIDI
- MPEG Video Playback
- Text to Speech, Speech to Text
- Wireless Web Server

- http://www.research.digital.com/wrl/projects/Itsy/index.html

## IPic – Match Head Size Web Server

-  PIC 12C509A running @ 4MHz
- IPic Tiny TCP/IP stack
- HTTP 1.0 compliant web server
- Simple telnet server
- 24LC256 i2c EEPROM

- http://www-ccs.cs.umass.edu/~shri/iPic.html

## Are We Destined to Compute on Toys?

- U128-bit RISC chip running at 300 MHz
- RAM:  32MB DRAM
- Graphics Processor:  150 MHz, 4MB integrated VRAM
- Sound:  SPU2, 2MB RAM (AC3, Dolby Digital, DTS support)
- Drive:  4X DVD, 24X CD read speeds (PlayStation, DVD, audio CD support)
- Dimensions:  178mm by 301mm by 78mm (12 inches by 7 inches by 3 inches)
- Weight:  2.1 kg (4 lbs. 10 oz.)

##        The PSX2's Emotion Engine provides ten floating-point multiplier-accumulators, four floating-point dividers, and an MPEG-2 decoder to deliver killer multimedia performance. 

-        The PSX2's Emotion Engine provides ten floating-point multiplier-accumulators, four floating-point dividers, and an MPEG-2 decoder to deliver killer multimedia performance.

- Each vector unit has enough parallelism to complete a vertex operation (19 mul-adds + 1 divide) every seven cycles.

## Playstation2 Specs

## With 4M of multiported DRAM and 16 pixel processors, the 42.7-million-transistor graphics synthesizer is 16.7 * 16.7 mm in a 0.25-micron process with 0.25-micron gates. (Source: SCE) 

- With 4M of multiported DRAM and 16 pixel processors, the 42.7-million-transistor graphics synthesizer is 16.7 * 16.7 mm in a 0.25-micron process with 0.25-micron gates. (Source: SCE)

- The Emotion Engine, heart of Sony's second-generation PlayStation, implements 10.5 million transistors and measures 17 * 14.1 mm in a 0.25-micron four-layer-metal 1.8-V process with 0.18-micron gates. (Source: SCE and Toshiba)

## IRAM Vision: Intelligent PDA

- (UCB V-IRAM/ISTORE)

- Intelligent RAM project@ UCB
- (Patterson, Yelick et al.)
- One of multiple Processor in
- Memory Projects aimed at
- addressing the memory bottleneck
- by putting processors IN the memory.

## ISTORE Hardware Vision

- System-on-a-chip enables computer, memory, without significantly increasing size of disk
- 5-7 year target:

- MicroDrive:1.7” x 1.4” x 0.2” 	2006: ?
- 1999: 340 MB, 5400 RPM, 	5 MB/s, 15 ms seek
- 2006: 9 GB, 50 MB/s ?   (1.6X/yr capacity,1.4X/yr BW)
- Integrated IRAM processor
- 2x height
- Connected via crossbar switch
- growing like Moore’s law
- 10,000+ nodes in one rack!

- (UCB V-IRAM/ISTORE)

## Berkeley’s Vector IRAM

- Vector processing
- high-performance for media processing
- low power/energy for processor control
- modularity, low complexity
- scalability
- well understood software development (Cray Compilers)
- Embedded DRAM
- high bandwidth for vector processing
- low power/energy for memory accesses (< 2 WATTS)
- modularity, scalability
- small system size

- (UCB V-IRAM/ISTORE)

## Block Diagram

- (UCB V-IRAM/ISTORE)

## V-IRAM Design Overview

- 64b MIPS scalar core
- coprocessor interface
- 16KB I/D caches
- Vector unit
- 8KByte vector register file
- support for 64b, 32b, and 16b data-types
- 2 arithmetic (1 FP), 2 flag processing, 1 load-store units
- 4 64-bit datapaths per unit
- DRAM latency included in vector pipeline
- 4 addresses/cycle for strided/indexed accesses
- 2-level TLB

- Memory system
- 8 2MByte eDRAM banks
- single sub-bank per bank
- 256-bit synchronous interface, separate I/O signals
- 20ns cycle time, 6.6ns column access
- crossbar interconnect for 12.8 GB/sec  per direction
- no caches
- Network interface
- user-level message passing
- dedicated DMA engines
- 4 100MByte/s links

- (UCB V-IRAM/ISTORE)

## VIRAM-1 Floorplan

- DRAM
- Bank
- 0

- DRAM
- Bank
- 6

- DRAM
- Bank
- 4

- DRAM
- Bank
- 2

- DRAM
- Bank
- 1

- DRAM
- Bank
- 7

- DRAM
- Bank
- 5

- DRAM
- Bank
- 3

- Vector
- Lane 3

**M**
**I**
**P**
**S**

**N**
**I**

**I**
- O

- Vector
- Lane 0

- Vector
- Lane 1

- Vector
- Lane 2

**C**
**T**
**L**

- (UCB V-IRAM/ISTORE)

## Vector Lanes

**64b**

**256b**

**Control**

- (UCB V-IRAM/ISTORE)

## V-IRAM Prototype Summary

- Technology:
- 0.18um eDRAM CMOS process (IBM)
- 6 layers of copper interconnect
- 1.2V and 1.8V power supply
- Memory:	16 MBytes
- Clock frequency:	 200MHz
- Power: 2 W for vector unit and memory
- Transistor count: ~140 millions
- Peak performance:
- GOPS: 3.2 (64b), 6.4 (32b), 12.8 (16b)
- GFLOPS: 1.6 (32b)

- (UCB V-IRAM/ISTORE)

## Petaflops from V-IRAM?

- In 2005 .. V-IRAM 6 Gflops/64 MBs
- 50 TF per 19” Rack (~10K CPUs/Disk assemblies)
- 100 drawers of 100 processors (like library cards)
- cross bar interconnected at Nx 100 MB/s
- 20 optically interconnected Racks
-  1015 FLOPS ~$20M
- Power ~ 20K Watts x 20 = 400K Watts
- Would use mostly the same parts as Cell phones and PDAs
- Probably could have only four types of components

## The Computational Grid  Moving From Local to Global Access to Computing and Information Resources

## The Emerging Concept of a National Scale Information Power Grid

## The Grid Links People with Distributed Resources on a National Scale

**http://science.nas.nasa.gov/Groups/Tools/IPG**

## Examples of Grids Testbeds

- Applications oriented Gigabit Testbeds
- CASA, Aurora, BLANCA, etc.
- I-WAY (at SC95 in San Diego)
- Gusto ‘97 and ‘98,’99,  (Globus project testbed)
- UW and Microsoft HDTV over NTON Demo at SC99
- STARTAP iGrid demonstration effort at SC98
- NCSA Alliance National Technology Grid
- SDSC Integrated Metasystems (Legion Testbed)
- NASA Information Power Grid project
- DOE NGI Testbeds (e.g. EMERGE)

**National Computational Science Alliance, National Technology Grid: Charlie Catlett et. al., NCSA**

## The Vision for the Grid

- Persistent, Universal and Ubiquitous Access to Networked Resources
- Common Tools and Infrastructure for Building 21st Century Applications
- Integrating HPC, Data Intensive
- 	Computing, Remote Visualization
- 	and Advanced Collaborations
- 	Technologies

## The Grid from a Services View

- _Resource-specific implementations of basic services_

- :

- E.g., Transport protocols, name servers, differentiated services, CPU schedulers, public key

- infrastructure, site accounting, directory service, OS bypass

- _Resource-independent and application-independent services_

- :

- E.g.,

-  authentication, authorization, resource location, resource allocation, events, accounting,

- remote data access, information, policy, fault detection

- Distributed

- Computing

- Applications

- Toolkit

**Grid Fabric**

**(Resources)**

**Grid Services**

**(Middleware)**

**Application**

**Toolkits**

- Data-

- Intensive

- Applications

- Toolkit

- Collaborative

- Applications

- Toolkit

- Remote

- Visualization

- Applications

- Toolkit

- Problem

- Solving

- Applications

- Toolkit

- Remote

- Instrumentation

- Applications

- Toolkit

**Applications**

- Chemistry

- Biology

- Cosmology

- Nanotechnology

- Environment

## Midwest Networked CAVE and ImmersaDesk Sites

**Argonne**
**UIC-Chicago **
**UIUC-Urbana**
**U Wisconsin**
**U Michigan**
**Indiana U**
**U Iowa**
**Iowa State**
**U Minnesota**
**U of Chicago**

## Access Grid Nodes 

- Access Grid Nodes

- Access Grid Nodes Under Development
- Library, Workshop
- ActiveMural Room
- Office
- Auditorium

## CorridorOne  Distance Visualization

- CorridorOne  Distance Visualization

- Argonne
- Berkeley Lab
- Los Alamos
- Princeton
- University of Illinois
- University of Utah
- DOE NGI Project

## Acknowledgements

## Many Thanks to Our Research Collaborators

- Los Alamos (Ahrens, Painter, Reynders)
- Utah (Johnson, Hansen)
- Princeton (Li, Finkelstein, Funkhouser)
- UIUC (Reed, Brady)
- EVL/UIC (DeFanti, Leigh, Sandin, Brown)
- LCSE/Minnesota (Woodward)
- LBNL (Lucas, Lau, Johnston, Bethel)
- NCSA (Smarr, Catlett, Baker, Cox)
- Kitware (Schroeder, Lorensen)
- UCB (Demmel, Culler)

## More Thanks Owed For:

- AccessGrid and Metro
- NCSA, Utah, EVL/UIC, UIUC, Princeton, Maui, UNM, GATech, UKY, LANL, LBNL, Boston  (20+ by end of FY00)
- ActiveMural and MicroMural
- Princeton, UIUC, LLNL, Minn, EVL
- Immersive and Large-Format Scientific Visualization
- Chicago, NCSA, EVL/UIC, LANL, LSU, Kitware, Utah, UIUC
- ManyWorlds/CAVERNsoft
- NCSA Alliance, UIUC, LSU, APS GCA, Nalco, AVTC

## The FL Group at Argonne/Chicago is:

- Rick Stevens, Mike Papka, Terry Disz,  Bob Olson, Ivan Judson, Randy Hudson, Lisa Childers, Mark Hereld, Joe Paris, Tom Brown, Tushdar Udeshi, Justin Binns, Mary Fritsch,  Chenhui Yang
- Thanks to Argonne, UChicago, DOE and NSF for $$ support!!
- Thanks to the NCSA, LANL, LBNL, Princeton, Utah and EVL for pics and movies

## Slide 65

## Questions?

## Questions?

## Questions?

## Questions?

## Questions?

## Questions?

## Questions?

## Questions?

## Questions?

## Partnerships for Advanced Computational Infrastructure Partner and Leading Edge Sites

## Universities With Projects Using NSF Supercomputers

- 850 Projects in 280 Universities

## University High Speed NetworksNSF vBNS and Internet2 Abilene

## Prototyping America’s 21st Century Information Infrastructure

- The National Technology Grid

## The Next Step is the Integration of International Research Networks 

- www.startap.net

- The Next Step is the Integration of International Research Networks

- NSF Funded Interconnection Point to US Networks
- Managed by Alliance Sites EVL (UIC) and ANL
- Operated by Ameritech Advanced Data Services

**Applications:**
**  ****Remote Instrumentation**
**  ****Shared Virtual Realities**
**  ****Tele-Immersion**
**  ****Distributed Computing**
**  ****Real-Time Client/Server**
**  ****Interactive Multimedia**
**  ****Tele-Instruction**
**  ****Tele-Medicine**
**  ****Digital Video**

## A Telecommunications Trend 

- Data traffic will exceed voice traffic in 3-5 years
- Voice traffic will become an small fraction of total traffic within 10 years
- Unlimited POTS voice service will be tossed in as a perk
- Bandwidth prices will eventually begin to follow Moore’s law
- 2X capability every 18 months
- Eventually we will think of bandwidth as “free like the air”

## Slide 81

## The Vision for the Grid

- Persistent, Universal and Ubiquitous Access to Networked Resources
- Common Tools and Infrastructure for Building 21st Century Applications
- Integrating HPC, Data Intensive
- 	Computing, Remote Visualization
- 	and Advanced Collaborations
- 	Technologies

## The Grid from a Services View

- _Resource-specific implementations of basic services_

- :

- E.g., Transport protocols, name servers, differentiated services, CPU schedulers, public key

- infrastructure, site accounting, directory service, OS bypass

- _Resource-independent and application-independent services_

- :

- E.g.,

-  authentication, authorization, resource location, resource allocation, events, accounting,

- remote data access, information, policy, fault detection

- Distributed

- Computing

- Applications

- Toolkit

**Grid Fabric**

**(Resources)**

**Grid Services**

**(Middleware)**

**Application**

**Toolkits**

- Data-

- Intensive

- Applications

- Toolkit

- Collaborative

- Applications

- Toolkit

- Remote

- Visualization

- Applications

- Toolkit

- Problem

- Solving

- Applications

- Toolkit

- Remote

- Instrumentation

- Applications

- Toolkit

**Applications**

- Chemistry

- Biology

- Cosmology

- Nanotechnology

- Environment

## Teleimmersion Networking Requirements

-  Immersive environment
-  Sharing of objects and virtual space
-  Coordinated navigation and discovery
-  Interactive control and synchronization
-  Interactive modification of environment
-  Scalable distribution of environment

## Globus-Enabled Support forPriority Traffic on the Internet

- Experiments performed on GARnet, testbed constructed with Cisco support

## Active Spaces   Exploring Future Workspace Environments

- Related Work:
- Xerox PARC Ubicomp (Wieser et. al.) MIT Oxygen Project
- UIUC’s SmartSpaces (Reed et. al.)
- UNC’s Office of the Future (Fuchs et. al.)

## Organizational and Social Trends 

- Distribution and Virtualization of Organizations
- Merging of Work/play/home/learning
- Work As Conversation/communication/collaboration
- Emerging Dominance of Multidisciplinary Problems
- Emerging Dominance of Consumer Products/tech Drivers
- 	What are possible infrastructures and environments to support future work, learning, living and play?

## Active Spaces   Workspace Environments

- Ensembles of Display  and Computer Systems and Software Environments Cooperatively Working to Support the User
- New Application Environment Metaphor
- Above, Beyond, Behind and Below the Desktop
- Includes the Desktop and Personal Devices, but Also Includes Large-format, Immersive and Room Scale Systems
- Challenges:
- integrate these different modalities in positive ways to enable more rapid problem solving
- provide bridges to existing tools and environments
- combine with high-end visualization and collaboration systems

## StationOne

- Early Concept sketch for a
- single user visual SuperStation.
- Designed to be both a
- high-resolution visualization
- environment and a high-end
- node in a collaboratory for
- advanced simulation.

- Argonne Futures Lab Group  (Stevens et al.)

## Slide 90

## Proposed: Totally Active Work Space (TAWS)

- Electronic Visualization Lab (DeFanti et al.)

## Active Spaces  As Spaces

- Blending Workspace Architecture and Information and Technology Rich Environments
- How groups of people organize for doing complex tasks
- How space affects work/thinking
- How distance and time affects working
*Inspired by the concept of the Medieval Craft Workshop*

## Active Spaces  As Grid Nodes

- They support several key notions:
- Blending of virtual and physical interfaces
- Uses variety of display types from immersion to small 2D displays
- Uses variety of scales of interfaces
- Couple data resources to physical interfaces provides tangible handles or references to virtual resources
- Support scaling of streams and bandwidths to dozens of persistent real-time streams
- Support the integration of many types of tools and components allows for ad hoc integration of new and exiting tools

## Midwest Networked CAVE and ImmersaDesk Sites

**Argonne**
**UIC-Chicago **
**UIUC-Urbana**
**U Wisconsin**
**U Michigan**
**Indiana U**
**U Iowa**
**Iowa State**
**U Minnesota**
**U of Chicago**

## Welcome to Chiba City

## Slide 96

## Advanced Display Environments

- Advanced display systems provide more resolution and capability (e.g. Immersion, Stereo) than the desktop
- ActiveMural
- MicroMural
- CAVE
- ImmersaDesk
- The Futures Lab Group is evaluating high-end scientific visualization applications in all of these environments
- We are beginning to study which types of devices are appropriate for which types of datasets, analysis goals and visualization modalities

## Slide 98

## Slide 99

## Directions for Active Space Development

- We Are Constructing Experimental Distributed Intentional Workspaces Based on the Concept of the “Artists Workshop or Studio” As Imagined in Medieval Times but Updated for the 21st Century
- We are Focusing on Three general Problems:
- Exploration of Advanced Display Environments
- Group to Group Collaboration Technologies
- Grid based Interaction and Visualization Technologies, Architecture Systems and Infrastructures
- Computational Science,  High-end Scientific Visualization , and Parallel Systems Design and Development and are our Drivers

## Active Mural  Beyond the MegaPixel

- Related Work:
- Stanford's Info Mural (Hanrahan, et. al.)
- Princeton’s Giant Display Wall (Li, et. al.)
- MIT AI Lab’s Big Inexpensive Display (Knight et. al.)
- Minnesota’s Great Wall of Power (Woodward et. al.)
- EVL’s Infinity Wall (DeFanti, et. al.)
- UIUC’s SmartSpaces Wall (Reed et. al.)
- UNC’s Office of the Future (Fuchs et. al.)

## ActiveMural, a Tiled Display Wall

- Argonne, Princeton and UIUC Collaboration
- 8’ x 16’ display wall
- Jenmar Visual Systems BlackScreenTM technology, > 10000 lumens
- 8 LCD  15 DLP  24 DLP
- 8-20 MegaPixels
- SGI and Linux drivers
- VR and ActiveSpace UI

## Slide 103

##  The Active Mural Lab with Access Grid Tech

-  The Active Mural Lab with Access Grid Tech

## Corridor One  Attacking the Distance Visualization and Resource Problem

- Related Work:
- Argonne/EVL I-WAY Project (DeFanti/Stevens, et. al.)
- Argonne/ISI Globus Project (Foster/Kesselman et. al.)
- Darpa Gigabit Testbeds (Kahn et. al.)
- NCSA Metacomputing Project (Catlett/Smarr et. al.)
- NASA IPG Project (Johnston et. al.)

## CorridorOne  Distance Visualization

- CorridorOne  Distance Visualization

- Argonne
- Berkeley Lab
- Los Alamos
- Princeton
- University of Illinois
- University of Utah
- DOE NGI Project

## Data Servers
Analysis and Manipulation Engines
Visualization Servers
Visualization Clients
Display Device Interfaces
Advanced Networking Services

- Data Servers
- Analysis and Manipulation Engines
- Visualization Servers
- Visualization Clients
- Display Device Interfaces
- Advanced Networking Services

- C1 Distance Corridor Systems Architecture

## Our Belief and Conclusion

- 	The Development and Success of  Future Integrated Networked Digital Group Workspaces Will Be Strongly Influenced by a Number of Technologies, Concepts and Issues:
- The Emergence of the Grid and The Widespread Use of Commodity Systems and Components
- Group Oriented Collaboration Environments and Tools
- Increasing Integration of Display Systems and Internetworked Digital Devices Into the Environment, Especially Wireless Devices and Large-Format  (non-Desktop) Displays
- New Visual and Interactive Modalities (e.g. Teleimmersion) Will Emerge to Address Issues of Working over Distance and Time
- These New Types of Emerging Environments Needs to Support Both Formal and Informal Interactions
- Informal Social Use of These Environments Is Critical to Their Use, Adoption and Eventual Success
- How People Work in These New Spaces Must Be Studied and the Impacts on Productivity, Learning and Development Must Be Learned

## ActiveMural, Mural: Price Performance

*Active*Mural, Mural: Price Performance

- Leverages COTS Technology
- Commodity Projectors Cost ~ $0.006/pixel
- High End Projectors Cost ~ $0.015/pixel
- Platform Independent
- SGI, NT, Linux Multiple Configurations
- Complete Linux Based System < $100K
- Screen and Frame ~10K
- Projectors ~36K
- Computers W/ Graphics ~40K
- Collaboration With Princeton and UIUC
- Coordinating Software Efforts
- Projector Mount Fabrication
- Virtual Framebuffer Software
- Diagnostics and Analysis Software

- Sub mm pixels

- Image blending

## The Mural

- The Mural

- Portable Six Projector Tiled Display for
- High-Resolution Visualization

## Slide 111

## Projector Alignment/ Image Blending

## Slide 113

## Slide 114

## Slide 115

## Slide 116

## Research Directions for ActiveMural, Mural

- Continued development and refinement of *Active*Mural
- Characterize and improve image quality (blending, filters, etc.)
- Improve calibration, reconfigurability and tuning
- Software environment (e.g. virtual frame buffers)
- Investigate use of *Active*Mural as multiresolution testbed
- Human factors studies and comparisons with CAVE and Idesks
- Development of smaller versions of AM (Mural) that can can be more widely deployed
- Six panel version under construction
- will be demonstrated today and at SC99
- StationOne prototype planning
- Office friendly version of Mural

## Access Grid  Integrating Group to Group Collaboration and Visualization

- Related Work:
- Berkeley and LBNL’s Mbone Tools (Jacobson et al.)
- Xerox PARC MOO and Jupiter  Projects (Curtis, et. al.)
- Argonne/NEU’s Labspace Project (Evard/Stevens et. al.)
- EVL’s  CAVERsoft (DeFanti, et. al.)
- UNC’s Office of the Future (Fuchs et. al.)
- DOE’s Collaboratory Pilots (Zalusec et. al.)

## Group-to-Group Interaction is Different

- Large-scale Scientific and Technical Collaborations Often Involve Multiple Teams Working Together
- Group-to-group Interactions Are More Complex Than Than Individual-to-individual Interactions
- The Access Grid Project Is Aimed at Exploring and Supporting This More Complex Set of Requirements and Functions
- The Access Grid Will Integrate and Leverage Desktop Tools As Needed

## Access Grid Nodes 

- Access Grid Nodes

- Access Grid Nodes Under Development
- Library, Workshop
- ActiveMural Room
- Office
- Auditorium

## Components of an Access Grid Node

## Slide 122

## Slide 123

## Slide 124

## The Access Grid is More Than  Teleconferencing 

- The Access Grid is More Than  Teleconferencing

- Physical Spaces Designed to Support Distributed Groupwork
- Multiple Virtual Collaborative Venues
- Agenda Driven Scenarios and Sessions
- Integration With GRID Services

## The Five Stages of Joint Work

- Awareness
- Interaction
- Cooperation
- Collaboration
- Organization

## Group-to-Group Requirements 

- Need to Enable Both Subgroup and Group Communication
- Support Multi-layered and Simultaneous Interactions
- Multiple Private/group-private/back Channels
- Integration of Groupware Tools/concepts With Collaboration

## The Docking Concept for Access Grid

- Private Workspaces - Docked into the Group Workspace

## The Illinois “True Grid” Dark Fiber Infrastructure 

- State Funded Network Testbed ($4.0M/FY00)
- Application Oriented
- QoS and Resource Scheduling
- Research and Development
- Network Research
- Advanced Systems Interfaces
- Technology Application
- Experimental Protocols
- Abilene/SuperNet Access for Chicago
- Possibility of a POP at ANL
- Access to MREN/AADS
- Technology Partners
- CISCO
- Fore Systems

## Proposed Terabit/s Midwest Network Testbed

- Includes NCSA, Argonne, U of Okla
- 10 Gbit/s Application Testbed Planned
- Industry partners include Williams Communications, Nortel, Corvis, Cisco
- Integration of GRID middleware with network QoS and network provisioning systems

- _Corvis All-Optical Technologies_
- 400 Gbit/s Transport per fiber strand
- Transmission distance to 3200 km
- 2.4 Tb/s optical switching