Bright Cluster Manager
Advanced HPC cluster management made easy

株式会社ベストシステムズ
代表取締役 西 克也

The Commonly Used “Toolkit” ApproachMost HPC cluster management solutions use the “toolkit”approach (Linux distro + tools)IIssues with the “toolkit” approach:







A
E
DExamples: Rocks, PCM, OSCAR, UniCluster, CMU, etc.
Tools typically used: Ganglia, Cacti, Nagios, Cfengine, System Imager,xCAT, Puppet, Cobbler, Hobbit, Big Brother, Zabbix, Groundwork, etc.

Tools rarely designed to work together
Tools rarely designed for HPCTools rarely designed to scale
Each tool has its own command line interface and GUI
Each tool has its own daemon and database
Roadmap dependent on developers of the tools

DA
B

Making a collection of unrelated tools work together


Requires a lot of expertise and scripting
Rarely leads to a really easy-to-use and scalable solution2

About Bright Cluster Manager

Bright Cluster Manager takes a much more fundamental& integrated approachDesigned and written from the ground up
Single cluster management daemon provides all functionality
Single, central database for configuration and monitoring data
Single CLI and GUI for ALL cluster management functionality

Which makes Bright Cluster Manager …


Extremely easy to use
Extremely scalable
Secure & reliable
Complete
Flexible
Maintainable3

Architecture

CMDaemon4

Bright Cluster Manager — Elements
Cluster Management GUI

Cluster Management Shell

SSL / SOAP / X509 / IPtables
Cluster Management Daemon

Provisioning

PBS ProTorqueMaui/MOAB
Grid Engine
SLURM
LSF

Monitoring
Automation
Health Checks
Management

Compilers
Libraries
Debuggers
Profilers

SLES / RHEL / CentOS / SL
SLES / RHEL / CentOS / SL
ScaleMP vSMP

5

Management Interface
Graphical User Interface (GUI) Offers administrator full cluster control
 Standalone desktop application
 Manages multiple clusters simultaneously
 Runs on Linux, Windows, MacOS X*
 Built on top of Mozilla XUL engine
Cluster Management Shell (CMSH)
 All GUI functionality also available through
Cluster Management Shell
 Interactive and scriptable in batch mode

Admin CLI7Node Provisioning
Image based
 Regular node image is a directory on the head node
 Unlimited number of images can be created Software changes for the regular nodes are made inside theimage(s) on the head node Provisioning system ensures that changes are propagated to theregular nodes

Nodes always boot over the network

Regular nodes PXE boot into Node Installer, which
Identifies node (switch port or MAC based)
Configures BMC
Partition disks (if any) and creates file systems (if needed)
Installs or updates software image (if needed)
Pivot the root from NFS to the local file system

12Architecture — Monitoring

CMDaemonBMCBMCBMC14

Workload Manager Integration
 Automatic installation
 Automatic configuration Sampling, analysis and visualization of workload
manager statistics
 Consistent GUI, User Portal and CLI front-end toworkload manager
 Bright cluster SOAP API provides consistent access towhole cluster, including workload manager
 Failover of workload manager
 Health checking
18

Cluster Health Management
 Goal: provide problem free environment for running jobs
 Four elements
1. Cluster management automation
2. Regular health checks– Actions that return PASS, FAIL or UNKOW N– Can be associated with a settable severity and a message– Can launch an action based on any response value

3. Prejob health checks– Let the workload manager hold the job very briefly– Check the health of each reserved node
– If unhealthy, take the node offline, inform the system administrator– Let the workload manager reschedule the job to a different set of nodes

4. Hardware stability & performance tests– Very wide range of tests– May include disk overwrites and reboot(s) All elements above are configurable and extensible
20

Bright Cluster Manager for GPGPU
 CUDA & OpenCL redistribution rights
 Current and previous versions of CUDA & OpenCL Easy switching between CUDA & OpenCL versions
 CUDA driver automatically compiled at boot time Support for all NVIDIA GPUs

21

The Future
Cloud bursting I

node004
node007

node005
node006

node001

node002

node00323

The Future
Cloud bursting II

node001

node003

Head node
node00224

The Bright Advantage
Productivity & Efficiency
1. Easy to learn and use
2. Installation in less than 30 minutes
3. Full insight in and control over the cluster
4. All elements of the cluster are managed (servers, switches,networks, etc.)
5. Flexible provisioning (incremental, live, diskfull, diskless, IB-only,
node discovery)
6. Comprehensive monitoring (graphs & rackview)
7. Powerful automation (thresholds, alerts, actions)
8. Vendor-independent workload manager integration
9. Integrated application development environment
10. Multi-cluster functionality
11. Easy, automatic updating from Linux & Bright repositories
12. Comprehensive GPU support
13. Rapid SMP deployment

25

The Bright Advantage
Uptime1. Built-in support for unattended, reliable head node failover
2. Comprehensive cluster health checking framework
3. Powerful burn-in environment
Performance
1. Single light-weight daemon
2. Daemons are optimized and synchronizedCompliance & compatibility
1. Intel Cluster Ready
2. Audited by DICE and several customer (e.g. DoD, Pharma’s)
3. Based on standard Linux distributions and kernels
4. Drivers included for most major hardware brands
5. Tried and tested for full compatibility with many ISV applications
26

The Bright Advantage
Scalability
1. Off-loadable provisioning
2. Efficient collection and processing of monitoring metrics
3. Tried & tested on largest clusters in the world
Security
1. Automated security and other updates from PGP signed
repositories
2. All internal + external communication encrypted using
public/private key encryption through SSH/SSL
3. Authentication based on X509 certificates
4. Role-based access control
5. Auditing of all administrator write actions
6. Firewalls
7. Secure LDAP27