# Presentation: 528323

## NCCS Network Roadmap

- Josh Lothian
**High Performance Computing Operations****National Center for Computational Sciences**

## NCCS network roadmap

**2007**
**Scaled Ethernet to meet wide-area needs and current day local-area data movement**
**Developed wire-speed, low-latency perimeter security to fully utilize 10Gb production and research WAN connections**
**Began IB LAN deployment**
**2008**
**Continue building out IB LAN infrastructure to satisfy the file system needs of the Baker system**
**Test Lustre on IB/WAN for possible deployment**
**Increase security on 10Gb WAN connections**

**Summary: Hybrid Ethernet/Infiniband (IB) network to provide both high-speed wide-area connectivity and uber-speed local-area data movement**

## NCCS network roadmap summary

**Ethernet**
**[O(10Gb/s)]**

**Ethernet core scaled to match wide-area connectivity and archive**

**Infiniband core scaled to match central file system and data transfer**

**Gateway**

**Viz**

**High-Performance**
**Storage System**
**(HPSS)****‏**

**Jaguar**

**Lustre**

**Baker**

**Infiniband**
**[O(100 Gb/s)]**

## NCCS WAN overview

**Juniper T640**

**Ciena CoreStream**

**Juniper T320**

**Force10 E600**

**Cisco 6509**

**NSTG**

**TeraGrid**

**CCS**

**DOE UltraScience/**
**NSF CHEETAH**

**CCS**

**OC-192 to TeraGrid**
**2 x OC-192 to USN**
**OC-192 to CHEETAH**

**OC-192 to Internet 2**
**OC-192 to Esnet**
**Internal 10Gb Link**
**Internal 1Gb Link**

**Ciena CoreDirector**

## NCCS Network 2007

**4**

**4**

**4**

**20**

**32**

**48**

**48**

**48**

**48**

**48**

**48**

**48**

**48**

**48**

**48**

**20 DDR**

**Infiniband**

**6-12 DDR**

**64 DDR**

**24 SDR**

**87 SDR**

**6-12 DDR**

**24-48 SDR**

**32 DDR**

**32**

**Ethernet**

**32 DDR**

**Spider30**

**Spider10**

**Viz**

**HPSS**

**Devel**

**E2E**

**Force10 E1200**

**Voltaire ISR2012**

**Cisco 7024D**

**Voltaire ISR9288**

## NCCS IB network 2008

**50 DDR**

**24 SDR**

**20 DDR**

**300 DDR**
**(50 DDR/link)****‏**

**64 DDR**

**32 DDR**

**87 SDR**

**Jaguar**

**Viz**

**HPSS**

**E2E**

**Devel**

**Spider240**

**Baker**

**50 DDR**

**48-96 SDR**
**(16-32 SDR/link)****‏**

**Spider10**

**Spider30**

**20 DDR**

**32 DDR**

**Voltaire ISR2012**

**Voltaire ISR9288**

**Cisco 7024D**

## 2007 milestones

**April 2007—Completed 10 G WAN upgrade**
**June 2007—Completed VL/QOS testing**
**November 2007—Complete fiber IB cable testing**
**December 2007—IBoIP/SDP decision point**

## 2008 milestones

**January—4x QDR testing**
**February—Deploy flow-based perimeter security**
**February—Complete 4X QDR testing**
**March—Enable routing of XT3 HSN to production network**
**June—Deploy redundant links between IB switches**

## IB over WAN testing

**Placed 2 x Obsidiam Longbow devices  between Voltaire 9024 and Voltaire 9288**
**Provisioned loopback circuits of various lengths on the DOE UltraScience Network and ran test**
**RDMA Test Results:**
**	****Local: 7.5 Gbps (Longbow to Longbow)****	****	****	**
**    ****ORNL ****↔**** ORNL (0.2 mile): 7.5 Gbps**
**	****ORNL ****↔**** Chicago (1400 miles): 7.46 Gbps****	****	**
**	****ORNL ****↔**** Seattle (6600 miles): 7.23 Gbps****	****	**
**	****ORNL ****↔**** Sunnyvale (8600 miles): 7.2 Gbps**

**20**

**81**

**Obsidian**
**Longbow**

**Ciena**
**CD-CI**
**(SNV)****‏**

**Ciena**
**CD-CI**
**(ORNL)****‏**

**Obsidian**
**Longbow**

**Voltaire**
**9024**

**Voltaire**
**9288**

**DOE UltraScience Network**

## Contact

- Josh Lothian
- High Performance Computing Operations
- National Center for Computational Sciences
- (865) 241-5563
- lothian@ornl.gov

-    Lothian_Networking_SC07