Ever Forward:
Lustre at Indiana University
Lustre User Group, Japan
November 2, 2017
Stephen Simms
Manager, High Performance File Systems
ssimms@iu.edu
Indiana University

Greetings from Indiana!

And Indiana University

Indiana University has
8 Campuses
In 2016 Total Student Count 114,160
IU has a central IT organization• UITS• 1200 Full / Part Time Staff• Approximately $160M Budget• HPC resources are handled by
Research Technologies division

Lustre at Indiana University

Multi-Resource Workflow
Data Source

Researcher's
Computer

Visualization
Resource

ComputeResource #1

ComputeResource #2

TapeArchivehttps://www.flickr.com/photos/davesag/18735941

The Idea in 2005

2006 Data Capacitor• First implemented in 2006
• NSF MRI Grant• 535 TB – DataDirect Networks S2A9550• Site-wide file system with clients in Bloomington and Indianapolis
spanning 80 km• Partnership between Dell and Data Direct Networks• 24 OSSs x 10 Gb/s

Lustre is scalable – 55 PB at LLNL

http://www.flickr.com/photos/shadowstorm/404158384/

Lustre is fast – 1 TB/s at ORNL

http://www.flickr.com/photos/dvd5/163647219/

Lustre can support thousands of clients

https://www.flickr.com/photos/gogrimm/3525598147

Expanding Lustre’s Reach2006 Single 10Gb Client Tests
977 MB/s between ORNL and IU (95.3%)
Using 10Gb TeraGrid connection
Dell 2950 client
2x dual 3.0 GHz Xeon
4GB RAM
1 Myricom Myri10G card
Ethernet mode
Lustre 1.4.7.1
16 (of 24) Data Capacitor OSSs

2007 Bandwidth Challenge Idea
Use the Data Capacitor across 2000 miles
Build a second small one in Reno, Nevada
Show science that RT supports every day• We support a diverse community • Acquisition and Visualization (IU)
• Live Chemistry Instrument Data
Saturate a single 10Gb connection• Rare Archival Material (RIT)
Use a production network• Humanities• Connected - I2, NLR, TeraGrid• Acquisition, Analysis, and Visualization• Trace Data (TUD)• Computer Science• Simulation Data (IU)
• Life Science• High Energy Physics

2007 Bandwidth Challenge Configuration

Bandwidth Challenge Results

2008 - Data Capacitor WAN
Goals:• Provide centralized short-term storage• Enable Workflows• Harness distributed resources• Push beyond a demo
360TB of DataDirect Networks S2A9550
4 OSS x 10 Gbit
Rudimentary UID Mapping for Lustre 1.6

Gas Giant Planet ResearchVisualization
Resource

PSC
Pittsburgh, PA
410 miles

DataCapacitor
WAN

NCSA
Urbana, IL
147 miles

Tape
Archive
MSU
Starkville, MS
607 miles

From 10Gb to 100 Gb Ethernet

2010 Partnership with Dresden Germany
100GbE

100GbE

100GBit/s Lambda
60km Dark Fiber

17*10GbE

17*10GbE

16*8 Gbit/s FC
5*40 Gbit/s QDR IB

16*20 Gbit/s DDR IB

16*8 Gbit/s FC

Performance from Dresden to Freiberg(60km)

Unidirectional Lustre: 11.79 GByte/s (94.4%)
TCP/IP: 98.5 Gbit/s (98.5%)
Link: 100 Gbit/s (100.0%)

SC11 Demonstration• Internet2 and ESnet, 50.5 ms RTT

SC11 Equipment – Seattle to Indianapolis

SC 11 Performance – Limited by Time
Measurement Efficiency

Test Time

Link

50.5 ms

4 hours

TCP iperf

96 Gb/s96%

6 hours

LNET

9.4 GB/s

77%

2 hours

IOR

6.5 GB/s

52%

2 hours

Applications

6.2 GB/s

50%

1 hour2012-2017 UID Mapping and Shared Key Code
Software Project Sponsored by OpenSFS
Executed by IU with support from Intelnodemap feature
UID Mapping – Spans Administrative Domains
Updates rudimentary MDS only UID mapping of DC-WAN
shared key feature
Allows secure connection between Lustre clients and serversBoth features are currently available in Lustre 2.10

In Use Today and Tomorrow2012 - Data Capacitor II• 5.3 PB formatted• DataDirect Networks SFA12K Hardware• 16 Object Storage Servers (96 GB RAM)
• 2 Metadata Servers (192 GB RAM)• High Throughput• ~20 GB/s via Ethernet• >40 GB/s via InfiniBand- 10Gb Ethernet systems: Karst, Carbonate Clusters- FDR InfiniBand systems: Big Red II and II+ Cray Machines- Currently running Lustre 2.5

2016 - Data Capacitor WAN 2Indiana University’s first attempt using ZFS and Lustre
Backed by 1.1 PB DataDirect Networks SFA-10K
Running Lustre 2.8
4 OSS nodes
40 Gb Ethernet
Project Focus / No Scratch Space
Uses Zpools and Lustre pools to create containers

2016 - Data Capacitor RAM• A solid state Data Capacitor (all non-system drives are solid state)• 35 TB of SSD on Lustre 2.8.0 backed by LDISKFS on HP Servers• 6 OSS/12 OSTs
• Each OST is a RAID-0 array of 4 x 800 GB Intel DC 3500 SSDs
• 2 active MDS supporting DNE2
• Each MDT is RAID-0 array of 4 800 GB Intel DC 3510 SSDs
• We are currently using DNE1 because of Lustre 2.5.1 Cray clients• 40 Gbps FDR-10 IB to HPC systems

2017-2018 - Slate / Slate Condo
2 New File Systems
Persistent Storage for Researchers
Slate - 4 PB
Slate Condo - 8 PB
Will Run ZFS
Will Run Lustre 2.10
Working with DataDirect Networks

Acknowledgments
Thanks to•••••

DDN for a gracious invitation and ongoing partnershipIntel for their continued support
Michael Kluge and Technische Universitat Dresden
Robert Henschel, Director of Science Community Tools at IU
The High Performance File System Team at IU

Thank You for Your Time!
Questions?