ADAC8

Cray Shasta Orchestration for HPC and Cloudjsparks@cray.com© 2019 Cray, a Hewlett Packard Enterprise company

Agenda• Overview• Scheduling• Traditional HPC• Cloud Orchestration• Emerging Technologies• Workflow agents• Capsules• Q&A© 2019 Cray, a Hewlett Packard Enterprise company

2• Objective: execute traditional batch jobs
and cloud-native services & workflows• Fully utilize system resources• Three components

User models

Overview
services

workflowjobs

HPC

DataSciencebatch• Scheduling services• Execution services© 2019 Cray, a Hewlett Packard Enterprise company

Platform Services• User models
SchedulingExecution

3

HPC Orchestration User Cases• Kubernetes Use Cases (for HPC)
• Projects need to express their workflows and supporting services in a portable
cloud-native way• Third-party tools will increasingly assume compatibility with the kubernetes
platform• Emerging workflow tools assume kubernetes or strongly support it• Argo: container native workflow engine for parallel jobs
• Kube-batch: batch ‘like’ scheduler for kubernetes• Others: kubeflow, nextflow, airflow, volcano© 2019 Cray, a Hewlett Packard Enterprise company4SCHEDULING© 2019 Cray, a Hewlett Packard Enterprise company

5Scheduler Technologies
Meta-schedulers• Mesos, …(GridWay)• Attributes:••

HPC Schedulers• WLM: Slurm, PBS, Flux,etc.
• Attributes:•••

Placement
Scheduling polices
Resource controls© 2019 Cray, a Hewlett Packard Enterprise company

Broker resources
Global resource view

Container Orchestrators• Kubernetes, Swarm,
Nomad, etc.
• Attributes:•••

Lifecycle Management
Monitoring
Placement

6Orchestration and Workload Management...

partitioned
Separateenvironments• Deploy containerized
applications separatefrom HPC
applications• Benefit: Avoid
disrupting existingenvironments• Challenge: Siloedpartitions© 2019 Cray, a Hewlett Packard Enterprise company

converged

Containers via a WLM

Orchestration job
scheduling features

Run both natively inshared environment• Use workload
manager toinstantiate containers• Use existingscheduling facilities inorchestrator• Orchestration andworkload manager
co-exist on the samecluster• Benefit: Containerized
workloads withminimal disruption totheir environment.

• Benefit: Leverageinnovation inorchestration tools• Challenge: no accessto orchestration
features• Challenge: Interaction
with traditional HPC
and relevantschedulers increases
complexity andpotentially decreases
performance• Benefit: With thisapproach, WLM acts
as a resourcemanager, making
resources available tothe Orchestrator.
• Challenge: Difficultyof sharing resources

7

Multi-framework Scheduling Options• A – Independent schedulers• Admin manages scheduling domains –fixed allocations• Rebalancing requires reboot• Inefficient use of system resources• C – Master worker scheduler• Corporative scheduling environment• Authoritative scheduling – singlesystem view
• Requires bridging master andsubordinate schedulers© 2019 Cray, a Hewlett Packard Enterprise company• B – Subordinate scheduler• User defined framework• Outer scheduler owns resources, Inner
uses resources• Independent/adversarial schedule policies• Inefficient use of system resources• D – Meta-scheduler environment• Supports multiple workload paradigms• Authoritative scheduling – single systemview• Requires scheduler modifications
8

Converged System
Domain Runtime
Environments

ResourceManagementHPC(MPI, SHMEM,...)© 2019 Cray, a Hewlett Packard Enterprise company

AI/ML/DL(Tensor Flow)

Global Resource Manager
Batch Schedulers(e.g. Slurm)

Resource Provisioning
Resources

Big Data Analytics(spark)Orchestrators(e.g. k8s)
OSCompute(e.g. processors, GPU)

Storage(e.g. POSIX, object)

Networking(e.g. Ethernet, IB)

9Assumptions in Orchestration Services•

Typically used in cloud environments – not HPC•

The user “owns” the resources•

Focused on deploying a set of services and keeping them in a good state••

Tends to assume there is always enough resource (or it can be quickly
provisioned – elastic consumption)•Persistent, long-running (no walltime)•Not HPC application focused☨
Definition of a consumable can be extensible☨ kube-batch offers some HPC semantics, job priorities, polices© 2019 Cray, a Hewlett Packard Enterprise company

Shasta Provisioned Stacks
Computes

Computes

k8s control
plane
k8s

application

Application
Slurmd

PALS

CRI

MoM

container

Shasta LinuxP1
HPC© 2019 Cray, a Hewlett Packard Enterprise company

Computes
Application
SlurmdP2
Analytics

PALS

CRI

MoM

container

Shasta LinuxShasta Linuxand

k8s control
plane
k8s

application

11orP3
Converged

11K8S – Extension Scheduler
job queue...

WLM Scheduler+sync with wlm++++nodes++

Services/workflow...

k8s Scheduler
slurmdresource filters

topology filters

priority filters

extender•CPU, RAM GPU•

Node utilization•Ports•Node affinity•

RAM, disk pressure•

Taint toleration•

Constraints•

Affinity•

Taints© 2019 Cray, a Hewlett Packard Enterprise company•kubelet

WLM decision/reservation

12

Shasta User View – Hybrid-Scheduling
NCN
UAI
HPC mod/sim

UAI

ComputeUAI

UAI

Analytics ML/AIUAIUAI© 2019 Cray, a Hewlett Packard Enterprise company

13

Scheduling Opportunities and Challenges• Job Type Characteristics• Parallel jobs – gang scheduling (kube-batch)• High throughput (million jobs) – scheduler latency• Interactive and batch• Resource Dependencies• Accelerators, networks, storage – resource plugins• Different Scheduling Options• On demand vs. queue• Workflow agents – argo, kube-flow, nextflow, etc.
© 2019 Cray, a Hewlett Packard Enterprise company

14

CAPSULES
Introduction© 2019 Cray, a Hewlett Packard Enterprise company

15

What are Capsules• A Capsule is a declarative specification of a desired runtime environment• Aims to provide a common abstraction for running a wide variety of workloads• Reduce cognitive overheads of running across differing schedulers andorchestrators• i.e. don't force end users to be K8S/Slurm/PBS experts• Focus has been on enabling interactive supercomputing© 2019 Cray, a Hewlett Packard Enterprise company

16Capsules – Design Goals• Declarative• Define what the user wants• Capsules handles details of launching that on the underlying platform• User Friendly• Inspired by common tools, such as conda, docker, git• Portable• Define a capsule once, run on any supported hardware/software (within reason)• Extensible• Extensible architecture allows vendor (and customers) to quickly support new functionality© 2019 Cray, a Hewlett Packard Enterprise company

17Capsules – Example ML Workflow> capsule create my-workflow> capsule open my-workflow> capsule add payload tensorflow> capsule edit payload add --config name simple-tf> capsule edit payload set --image default tensorflow/tensorflow> capsule edit payload remove --args "--user_str1=hello"> capsule edit payload add --config trainingScript simple.py> capsule edit payload set --data home `pwd` /home/trainer Directory false all> capsule close> capsule launch my-workflow> … wait for job to complete …
> capsule kill my-workflow• Open a capsule for editing, make changes and close it to save those changes• Then launch the capsule, wait for results and clean up when done
© 2019 Cray, a Hewlett Packard Enterprise company

18Capsules – Example HPC Workflow> capsule create mpi-example> capsule open mpi-example> capsule add payload hpc-job> capsule edit payload add --config command /lus/<username>/a.out> capsule edit payload add --resource instance nodes 4 \--resource instance per-node 4 --resource memory per-node 4G> capsule close> capsule launch mpi-example --attach> … wait for job to complete …
> capsule kill my-workflow• Very similar workflow to previous example, just different config options• This time using --attach to directly attach ourselves to the capsule• In this example this will result in monitoring the job output file© 2019 Cray, a Hewlett Packard Enterprise company

19

F O R WA R D L O O K I N GS TAT E M E N T S
This presentation may contain forward-looking statements that involve risks,uncertainties and assumptions. If the risks or uncertainties ever materialize or theassumptions prove incorrect, the results of Hewlett Packard Enterprise Company
and its consolidated subsidiaries ("Hewlett Packard Enterprise") may differ
materially from those expressed or implied by such forward-looking statements andassumptions. All statements other than statements of historical fact are statementsthat could be deemed forward-looking statements, including but not limited to anystatements regarding the expected benefits and costs of the transaction
contemplated by this presentation; the expected timing of the completion of thetransaction; the ability of HPE, its subsidiaries and Cray to complete the transaction
considering the various conditions to the transaction, some of which are outside theparties’ control, including those conditions related to regulatory approvals;
projections of revenue, margins, expenses, net earnings, net earnings per share,
cash flows, or other financial items; any statements concerning the expecteddevelopment, performance, market share or competitive performance relating toproducts or services; any statements regarding current or future macroeconomic
trends or events and the impact of those trends and events on Hewlett Packard
Enterprise and its financial performance; any statements of expectation or belief;
and any statements of assumptions underlying any of the foregoing. Risks,
uncertainties and assumptions include the possibility that expected benefits of thetransaction described in this presentation may not materialize as expected; that thetransaction may not be timely completed, if at all; that, prior to the completion of thetransaction, Cray’s business may not perform as expected due to transaction-related
uncertainty or other factors; that the parties are unable to successfully implement
integration strategies; the need to address the many challenges facing Hewlett
Packard Enterprise's businesses; the competitive pressures faced by Hewlett
Packard Enterprise's businesses; risks associated with executing Hewlett Packard
Enterprise's strategy; the impact of macroeconomic and geopolitical trends andevents; the development and transition of new products and services and theenhancement of existing products and services to meet customer needs andrespond to emerging technological trends; and other risks that are described in ourFiscal Year 2018 Annual Report on Form 10-K, and that are otherwise described orupdated from time to time in Hewlett Packard Enterprise's other filings with theSecurities and Exchange Commission, including but not limited to our subsequent
Quarterly Reports on Form 10-Q. Hewlett Packard Enterprise assumes no obligationand does not intend to update these forward-looking statements.

© 2019 Cray, a Hewlett Packard Enterprise company

20

QUESTIONS?

cray.com@cray_inc
jsparks@cray.com

linkedin.com/company/cray-inc-/