SUCCESS STORYFATTWIN SUPERSERVERSPOWER RUTGERSUNIVERSITY’S TOP RANKEDNEW SUPERCOMPUTER
Here’s a look at how Rutgers’ newsupercomputer ranks around the world:•2nd Among Big Ten universities•8th Among U.S. academic institutions•

49th Among global academic
institutions•

165th Among supercomputersworldwide

Source: TOP500

SUPERMICRO’S SUPERCOMPUTER CUSTOMER:RUTGERS UNIVERSITY
Ranked #2 among Big Ten Universities and #8 among U.S. Academic Institutions in
the Top500 List, the new supercomputer built with Supermicro FatTwin servers islocated at the Rutgers Discovery Informatics Institute (RDI2), New Jersey’s Center
for Advanced Computation. The institute uses advanced computation to address
today’s compute- and data-intensive grand challenges in science, engineering,and industry. The researchers there seek to broaden access to the state-of-the-art
computing technology that enables large-scale data analytics, computationalmodeling and simulations, all of which play an increasingly important role inmodern society.

October 2016
Ray Pang
Director of Solutions Marketing
Super Micro Computer, Inc.
980 Rock Avenue
San Jose, CA 95131 USA
www.supermicro.com

Intel Inside®. Powerful Data Center Outside.

FatTwin SuperServers Power Rutgers University's Top
Ranked New Supercomputer

CHALLENGES FOR RUTGERS• Rapid growing computing needs of its world-class faculty• Elevate the competitiveness of Rutgers researchers in the area ofBig Data and Big Science• Prohibitive costs of Supercomputing
Supercomputing costs can be prohibitive. Supercomputers can cost upwards ofseveral hundred million dollars in capital expenditures (CAPEX), and many millions
more in yearly maintenance, energy, and cooling operating expenditures (OPEX).
So when Rutgers set out to acquire a supercomputer to support the computing
needs of its world-class faculty, it utilized the most stringent criteria and a highly
competitive bidding process to select the best infrastructure vendor that provides
dramatic increase in computation power over current systems, greater speeds andexpanded storage capacity to faculty, researchers and students across Rutgers andthe state.

Figure 1. Supermicro® FatTwin™
SuperServer®

Supermicro® Hardware Portfolio•

560 FatTwin™ Compute Nodes•

20,160 cores•

140 TB of DRAM memory•218 TB of NVMe NVRAM•

677 TFLOPS peak performance•Intel® Omni-Path fabric•

Each FatTwin with dual Intel® Xeon®
Processor E5-2695 v4 CPUs, 256 GB ofRAM, and a 400 GB Intel NVMe driveThis HPC solution will significantly elevate the competitiveness of Rutgers
researchers in computational and data-enabled science, engineering andmedicine, as well as those in social science and humanities disciplines. The systemmanufacturer and integrator chosen after this rigorous bidding process is Super
Micro Computer Inc. of San Jose, California. With the majority of research anddevelopment efforts done in-house at its Silicon Valley headquarters, Supermicro
is in a unique position to build and deliver application-optimized solutions based
upon Rutgers’ requirements.

SUPERMICRO SOLUTIONS• Deployed high-density, high efficiency and high performance
multi-node FatTwin solution• One of the very first successful implementation of 100Gbps IntelOmni-Path Fabric• State-of-the-art rear door cooling for the best TCO
With a $10 million award from New Jersey Higher Education Leasing Fund, Rutgers
built a new supercomputer named “Caliburn,” using Supermicro’s state-of-the-art
FatTwin™ servers, the most powerful supercomputer system in the state.
The project was completed in three phases. Phase I, which went live in January
2016, provides approximately 150 TFLOPS of computational and data analytics
capabilities and one petabyte of storage to faculty and staff researchers throughout
the university. To date more than 100 faculty from 17 departments university
wide have signed up to use the system. Among the heaviest users have been
researchers in the Waksman Institute of Microbiology, the Department of Physics,
Department of Chemistry, and the Center for Integrative Proteomics Research.

Powered by Intel® Xeon® Processors

2FatTwin SuperServers Power Rutgers University's Top
Ranked New Supercomputer“As the leading provider of highperformance computing (HPC) solutions,Supermicro is very pleased to help enable
this state-of-the-art HPC solution atRutgers based on our multi-node FatTwin
architecture. Key features of these FatTwin
SuperServers include support for E5-2600
v4 processors, NVMe, 100 Gbps OmniPath fabric (OPA), and an innovative
cooling architecture to deliver maximum
performance while reducing the TCO for
the Caliburn supercomputer which will
supply high-performance computationaland data analytics capabilities toresearchers for years to come.”

Phase II construction included a new self-contained modular data center at Rutgers
University–New Brunswick. Phase III encompasses the installation of the Caliburn
supercomputer and final elements of the network, which provides high-speed
access to users.
The Supermicro solution is based on the FatTwin™ SuperServer system. The
supercomputer includes 560 FatTwin™ nodes, each node equipped with two Intel
Xeon E5-2695 v4 (Broadwell) processors, 256 gigabytes (GB) of DRAM, and a 400 GB
Intel NVMe drive. Overall, the system has 20,160 cores, 140 TB of DRAM and 218 TBof non-volatile memory. As a multi-node system, the FatTwin™ provides not only thedensity but also supports NVMe SSD that eliminates storage bottleneck and comeswith a PCI-E 3.0 x 16 expansion slot with enough bandwidth for 100Gbps OmniPath Architecture. With high node density for compute, NVMe SSD for fast storageand native 100Gbps OPA for throughput, the FatTwin™ represents the perfect
OPA Cluster Architecture
product for2:1Rutgers’
Caliburnsupercomputer. The standard performance level is
Fat-Tree Topology with Supermicro Cost-Effective 48-Ports ToR Switch Platform
603 TFLOPS with a peak performance of 677 TFLOPS.

Dr. Tau Leng
VP and GM of HPC
Super Micro Computer, Inc.

Intel OPA Switch
100 Series 48-Port
1 ISL
2 ISLs

#1

#19

#17

Intel OPA Switch
100 Series 48-Port

4 Nodes / 4U FatTwin
F628R3-RC1B+O“The Intel Omni-Path 100 Gb Architectureis specifically designed for large scale
HPC deployments such as Rutgers. As alarge scale deployment of the Omni-Pathtechnology, the integrated HPC solution atRutgers achieves high levels of scaling andpower efficiency, while demonstrating
the industry leading capability of OmniPath in large scale computing. As a leadpartner for Omni-Path, Supermicro’s
optimized product portfolio anddeployment capabilities were critical tobring the solution successfully online.”
John Beck
GM of the Intel Omni-Path Fabric Division
Intel Corporation

Powered by Intel® Xeon® Processors

8 Core Switches

6x 2U DP SuperServer
SYS-6028R-TR

8 Nodes / 4U FatTwin
F618R2-RTN+O

512 Nodes(Switch #1 - #16)

8 Nodes / 4U FatTwin
F618R2-RTN+O

19 Edge Switches

4 Head Nodes

6 Router Nodes

Exsiting FDR GPFS

48 Nodes (8 Future Nodes)(Switch #17 - #18)

560 Compute Nodes

Figure 2. Supermicro's Cost-effective HPC Architecture at Rutgers
To provide Rutgers with the best possible TCO, Supermicro professional service
team deployed the world’s most energy efficient and space-saving Coolcentric®
Rear Door Heat Exchanger solution. A passive liquid cooled heat exchanger that
replaces standard rear doors on IT racks, the Coolcentric® rear door cooling atRutgers uses 90% less energy than regular indoor cooling units and is 80% spaceefficient.

3FatTwin SuperServers Power Rutgers University's TopRanked New Supercomputer“We can train students on how use
next-generation technologies andcapabilities to better address problems
that are relevant to industry today,” Dr.
Parashar said. “That’s really what we’rehoping to do — build a whole co-op andinteractions of research and collaboration
around this instrument that we have sothat we can build better partnerships
moving ahead.”
Dr. ParasharDistinguished Professor of Computer Science
Rutgers University

Figure 3. Coolcentric Rear Door Water Cooling System is Deployed on Each RackRESULTS WITH SUPERMICRO• 2# Among Big Ten universities in the Top500 List• 8# Among U.S. academic institutions in the Top500 List
• 49# Among global academic institutions in the Top500 List

FOR MORE INFORMATIONSupermicro® FatTwin™ SuperServer®
www.supermicro.com/products/nfo/FatTwin.cfm
Rutgers Caliburn Supercomputerrdi2.rutgers.edu/caliburn
Intel® Xeon® Processor E5-2600 v4 Product
Familywww.intel.com/content/www/us/en/processors/xeon/xeon-e5-solutions.html
Intel® NVMe SSDs
www.intel.com/content/www/us/en/solidstate-drives/solid-state-drives-ssd.html

Powered by Intel® Xeon® Processors

Dr. Manish Parashar, distinguished professor of computer science at Rutgers andfounding director of the Rutgers Discovery Informatics Institute (RDI2), heads
the project. Dr. Parashar and Ivan Rodero, RDI2’s associate director of technicaloperations, designed the system with a unique architecture and capabilities basedon a new network interconnect developed by Intel (Omni-Path). The Rutgers
supercomputer is among the first clusters to use the Intel Omni-Path fabric and thelatest Intel processors.
Besides giving its scientists new power to rapidly solve complex calculations,Dr. Parashar, hopes to leverage its new capabilities by collaborating with private
industry to train the emerging workforce.
Along with users at Rutgers, the system will be accessible to researchers at otherNew Jersey universities and industrial users. RDI2 will work with the New Jersey
Big Data Alliance, which was founded by Rutgers and seven other universities inthe state, to build an industry users program. The capabilities of this new system
will establish New Jersey's reputation in advanced computing and benefit a broad
spectrum of industry sectors and academic disciplines.

4About Super Micro Computer, Inc.
Supermicro® (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology is a premier provider ofadvanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and Embedded
Systems worldwide. Supermicro is committed to protecting the environment through its “We Keep IT Green®” initiative and provides
customers with the most energy-efficient, environmentally-friendly solutions available on the market.
www.supermicro.com

The information contained in this document is subject to change without notice.
No part of this document covered by copyright may be reproduced in any form or by any means — graphic, electronic, or mechanical,including photocopying, recording, taping, or storage in an electronic retrieval system — without prior written permission of thecopyright owner.
Supermicro, the Supermicro logo, Building Block Solutions, We Keep IT Green, SuperServer, TwinPro™, TwinPro²™, SuperDoctor are
trademarks and/or registered trademarks of Super Micro Computer, Inc.
Ultrabook, Celeron, Celeron Inside, Core Inside, Intel, Intel Logo, Intel Atom, Intel Atom Inside, Intel Core,Intel Inside, Intel Inside Logo, Intel vPro, Itanium, Itanium Inside, Pentium, Pentium Inside, vPro Inside,Xeon, Xeon Phi, and Xeon Inside are trademarks of Intel Corporation in the U.S. and/or other countries.
© Copyright 2016 Super Micro Computer, Inc. All rights reserved.
Printed in USA

MKT-0006-09/2016-10-R6		

Please Recycle