Project: OpenEBS (Runtime / Cloud Native Storage)
File: OpenEBS Architecture and Design.pdf

v0.6 
February 28, 2018 
Architecture and Design 
Prerequisites 
●Kubernetes Concepts : Namespaces, RBAC,CRD, Taints & Tolerations, Pod Anti 
Affinity, etc., 
●Kubernetes Storage Concepts:  Running Stateful Workloads via PV, PVC, 
Storage Classes and Dynamic Provisioner 
●Kubernetes and CNCF Management Tools: Kube Dashboard, Prometheus, 
Grafana, Opentracing  & Jaeger 
●Kubernetes Incubator Projects:   Node Exporter, Node Problem Detector 
●Storage optimized for Containerized Applications 
●Stable, Secure and Scalable - Horizontally scalable to millions of Containers, Fault 
tolerant and Secure by default 
●Seamless integration into any private and public cloud environments. Vendor 
independent. 
●Non-disruptive software upgrades 
●Easy to setup. Low entry barrier. Developer and Operators Friendly. 
Design Goals and Constraints Persistent Volume Categories 
App
PVK8s Node 
App
PVApp
PV
Storage Server Vol Vol VolFilesystem / BlockDev NAS 
K8s Node 
App
PV
Vol VolApp
PV
VolApp
PV
Filesystem / BlockDev DAS 
K8s Node 
App
PVApp
PVApp
PV
Vol Vol Vol
Filesystem / BlockDev CAS 
NAS - Network Attached Storage (Example: GPD, EBS, Storage Appliances) 
DAS - Direct Attached Storage (Example: hostDir, Local PV) 
CAS - Container Attached Storage (Example: OpenEBS) 
Indicates functionality like replication, snapshots, encryption, compression, etc. Represent stateful Pods like Databases, etc. Container Attached Storage 
K8s Node 
App
PVApp
PVApp
PV
Vol Vol Vol
Filesystem / BlockDev CAS ○Storage Controllers (Targets) are running in 
containers. 
○These Storage Containers are orchestrated by 
Kubernetes, just like any other workloads. 
●Installation and Upgrades 
●Monitoring 
●Debuggability 
○Storage Containers are mainly dealing with: 
●Disk/Storage Management 
●Data - High Availability and 
●Data - Protection 
“OpenEBS is a CAS solution, that provides storage as a service to stateful workloads.  
OpenEBS hooks-into and extends the capabilities of Kubernetes to orchestrate 
storage services (workloads)” Kubernetes Cluster node2 
Filesystem / BlockDev 
node1 
Filesystem / BlockDev Using OpenEBS Volumes 
Pod
Stateful Workload 
(DB, etc) 
ov-ctrl 
(iSCSI Target) 
ov-rep-1 
(replica) ov-rep-2 
(replica) OpenEBS Volume 
Setup OpenEBS 
(StorageClasses 
StoragePools, Disks) Application 
(Deployment, PVC) 
(iSCSI PV) 
(iSCSI Initiator) OpenEBS Components 
Kubernetes Cluster kube-dashboard Cluster 
Admin 
kube-apiserver 
kube-heapster kube-etcd 
prometheus-maya grafana-maya 
maya-apiserver kubelet (n) 
maya-provisioner maya-nodebot (n) node_exporter (n) OpenEBS SC and CRD (StoragePools, VolumePolicies) (DaemonSets) 
The source code for the control plane components is located in mainly openebs/maya  repository. Developer 
or User 
(Deploy 
Workload) 
OpenEBS PVCs *OV Pods (m) *Workloads(m) Node Storage Management 
K8s-node can be configured with Persistent Storage  
as:
- Additional space on OS directory 
- NVMe Disks 
- SSD Disks 
- SAS Disks 
- SAN/NAS volumes 
- Object Store mounts 
OpenEBS Volume Replica ( OV Replica ) is a container 
running within a K8s Pod.  OV Replica will be granted 
access to local storage on the K8s-node using Local PV 
Options like: 
- hostDir 
- Block Disks by mounting (/dev) * 
 
OpenEBS Control Plane  manages the discovery, 
allocation to the OV Replica’s and monitoring of the 
storage attached to the node. 
*https://github.com/kubernetes/kubernetes/issues/58569 
*https://docs.google.com/document/d/1fG-KwUQNsuPYY40By
oBFqKJKpxzgyk7cQ5gqsGRXxfk/edit 
OV Replica 
PV(s) 
k8s-node 
Config 
MapOpenEBS - CRDs (extended Schema) 
OpenEBS Control Plane  uses Kubernetes etcd cluster to store the configuration information about 
OpenEBS related objects. 
While K8s primitives like PV, PVC, SC are used, they can in-turn refer to OpenEBS specific objects for 
further details. 
Some of the OpenEBS objects (CRDs) created by the OpenEBS Operator are: 
-Persistent Disk 
-Storage Pool 
-Storage Pool Claim 
-Persistent Volume Policy 
In addition to the above generic CRDs used by the openebs control plane, there can be custom CRDs 
specific to the Storage Engines like cStor: 
-cStorPool 
-cStorVolume 
Storage Schema - PersistentDisk 
NVMe SSD 
 SAS Disks HostDir SAN/NAS S3 
 NVDIMM SSDs 
PersistentDisk (PD),  
logical representation 
of the underlying 
storage attached to a 
node. K8s node 
PD(s) kube-etcd maya-nodebot 
node-problem-detector DaemonSets/System Pods Storage Schema - StoragePools 
NVMe SSD 
 SAS Disks HostDir SAN/NAS S3 
 NVDIMM SSDs 
PersistentDisk (PD),  
logical representation 
of the underlying 
storage attached to a 
node. K8s node 
PD(s) kube-etcd maya-nodebot 
node-problem-detector DaemonSets/System Pods maya-cstor
-operator 
SPC(s) 
file/block SP(s) 
(type=file) cStor  Pool 
SP(s) 
(type=block) Storage Schema - OpenEBS Volumes 
(Replicas) 
NVMe SSD 
 SAS Disks HostDir SAN/NAS S3 
 NVDIMM SSDs 
PersistentDisk (PD),  
logical representation 
of the underlying 
storage attached to a 
node. K8s node 
PD(s) kube-etcd maya-nodebot 
node-problem-detector DaemonSets/System Pods maya-cstor
-operator 
SPC(s) 
file/block SP(s) 
(type=file) cStor  Pool 
SP(s) 
(type=block) OV Replica 
(jiva) 
PV
(hostdir) OV Replica 
(block) OV Ctrl 
(jiva) OV Ctrl 
(cStor) maya-nodebot 
APIs (https / gRPC) 
Collector 
Discoverer Provisioner Monitor CLI (COBRA) 
kube-etcd prometheus-maya 
OS Utilities / Daemons kube custom controller and etcd cachestor Metrics Maya-nodebot runs on 
each node as a 
daemon-set. 
Has access to the 
underlying disk subsystem 
and can handle the disk 
add/lost events and can 
monitor for disk errors. 
Disk details and status 
(lost/deleted/healthy) are 
updated as CR into the 
etcd. 
Disks are monitored for IO 
stats and smart and 
metrics can be collected 
via prometheus. 
When critical events are 
observed (say disk lost 
from an active pool) calls 
maya-apiserver for 
handling or can call the 
Replica impacted. maya-apiserver 
APIs (https / gRPC) 
OV Provisioner CLI (COBRA) 
kube-etcd 
K8s Orchestrator Provider maya-provisioner 
kube-apiserver OV Snapshots 
OV Metrics OV Profiles 
maya-mulebot 
OV (replica) OV (target) prometheus-maya maya-provisioner 
APIs (https / gRPC) 
Create CLI (COBRA) 
kube-etcd 
K8s Dynamic Controller maya-apiserver 
kube-apiserver Metrics Update Delete prometheus-maya maya-volume-exporter 
APIs (https / gRPC) 
Collector CLI (COBRA) 
prometheus-maya OpenEBS Volume Exporter grafana-maya maya-mulebot (TBD) State Diagrams 
OpenEBS Volume - State Diagram 
does-not-exist initializing running deleting 
errored degraded 
offline migrating Sequence Diagrams 
(TBD) 
UI 
(Kubernetes 
Dashboard) 
TODO - Could list OpenEBS Volumes created using this Storage class Should include outstanding alerts  at the top of the page. Storage: 
avail/used 
On each 
pod. 
Should include Events at the bottom of the page.