Repository: apache/incubator-heron (language: python)

Issue:
Management/Driver pod should not use the same resources as the other topology pods. Should create separate deployment/service.
This ticket is to decouple the creation of the management pod and the executor pods (the pods that run the spouts and bolts). The first pod in the StatefulSet does not run any processing elements, it merely runs some management processes. The current deficiency with the way we schedule jobs in Kubernetes is that the first pod takes an equal amount of CPU and memory when compared to the executor pods. This can be wasteful when running a large number of topologies.

We want to allow for a separate Kuberenetes StatefulSet to manage the Manager Pod. This StatefulSet should be configured for a default CPU/Memory request/limit that can be overridden though submission time configuration.

The Managment StatefulSet and Executor StatefulSet should both support the new features we have recently added (i.e. dynamic PVCs, PodTemplate, dynamic labels, dynamic annotations)
[Heron-3723] Add support for Persistent Volumes for stateful storage
***Feature #3723: Add support for Persistent Volumes for stateful storage***

>  `PersistentVolumeClaims` and mount points similar to the feature found in [Spark](https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes).

> This will overwrite the `Volumes` and `VolumeMounts` provided in a custom Pod Template which clash with CLI input. Heron's entries will take precedence. This is to support on-topology-submit updates to the `Volumes` and `VolumeMounts`.

> `PersistentVolumeClaims` for dynamic volumes will be created in the same namespace as the Heron API Server. They *will* be removed once the Topology is killed.

> [Dynamic Volume provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) must be enabled on the K8s cluster.

> To disable the feature: `-D heron.kubernetes.persistent.volume.claims.cli.disabled=true`.

Added support for the following options:
* `storageClass`
* `sizeLimit`
* `accessModes`
* `volumeMode`
* `path`
* `subPath`

CLI Command:

`--config-property heron.kubernetes.volumes.persistentVolumeClaim.[VOLUME NAME].[OPTION]=[VALUE]`

Example:
Commands:
```bash
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.storageClassName=storageClassNameOfChoice
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.accessModes=comma,separated,list
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.sizeLimit=555Gi
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.volumeMode=volumeModeOfChoice
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.path=path/to/mount
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.subPath=sub/path/to/mount
```

Will generate the PVC:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nameOfVolumeClaim
spec:
  volumeName: volumeNameOfChoice
  accessModes:
    - comma
    - separated
    - list
  volumeMode: volumeModeOfChoice
  resources:
    requests:
      storage: 555Gi
  storageClassName: storageClassNameOfChoice
```

With Pod Spec entries for `Volume`:

```yaml
volumes:
  - name: volumeNameOfChoice
    persistentVolumeClaim:
      claimName: nameOfVolumeClaim
```

With `Executor` container entries for `VolumeMounts`:

```yaml
volumeMounts:
  - mountPath: path/to/mount
    subPath: sub/path/to/mount
    name: volumeNameOfChoice
```

---

**DESIGN:**

There is an enum in the `KubernetesConstants` class called `Kubernetes.PersistentVolumeClaimOptions` which contains the CLI options that are accepted.

Parameters from the CLI are parsed into the data structure:
`Map<String, Map<Kubernetes.PersistentVolumeClaimOptions, String>>`.
This maps the volume's name to its key-value pairs for its configuration.

Adding more options to the functionality simply requires the extension of the `PersistentVolumeClaimOptions` enum and adding the entries to the switch statement in `V1Controller.createPersistentVolumeClaims`.

[Heron-3723] Add support for Persistent Volumes for stateful storage
***Feature #3723: Add support for Persistent Volumes for stateful storage***

>  `PersistentVolumeClaims` and mount points similar to the feature found in [Spark](https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes).

> This will overwrite the `Volumes` and `VolumeMounts` provided in a custom Pod Template which clash with CLI input. Heron's entries will take precedence. This is to support on-topology-submit updates to the `Volumes` and `VolumeMounts`.

> `PersistentVolumeClaims` for dynamic volumes will be created in the same namespace as the Heron API Server. They *will* be removed once the Topology is killed.

> [Dynamic Volume provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) must be enabled on the K8s cluster.

> To disable the feature: `-D heron.kubernetes.persistent.volume.claims.cli.disabled=true`.

Added support for the following options:
* `storageClass`
* `sizeLimit`
* `accessModes`
* `volumeMode`
* `path`
* `subPath`

CLI Command:

`--config-property heron.kubernetes.volumes.persistentVolumeClaim.[VOLUME NAME].[OPTION]=[VALUE]`

Example:
Commands:
```bash
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.storageClassName=storageClassNameOfChoice
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.accessModes=comma,separated,list
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.sizeLimit=555Gi
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.volumeMode=volumeModeOfChoice
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.path=path/to/mount
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.subPath=sub/path/to/mount
```

Will generate the PVC:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nameOfVolumeClaim
spec:
  volumeName: volumeNameOfChoice
  accessModes:
    - comma
    - separated
    - list
  volumeMode: volumeModeOfChoice
  resources:
    requests:
      storage: 555Gi
  storageClassName: storageClassNameOfChoice
```

With Pod Spec entries for `Volume`:

```yaml
volumes:
  - name: volumeNameOfChoice
    persistentVolumeClaim:
      claimName: nameOfVolumeClaim
```

With `Executor` container entries for `VolumeMounts`:

```yaml
volumeMounts:
  - mountPath: path/to/mount
    subPath: sub/path/to/mount
    name: volumeNameOfChoice
```

---

**DESIGN:**

There is an enum in the `KubernetesConstants` class called `Kubernetes.PersistentVolumeClaimOptions` which contains the CLI options that are accepted.

Parameters from the CLI are parsed into the data structure:
`Map<String, Map<Kubernetes.PersistentVolumeClaimOptions, String>>`.
This maps the volume's name to its key-value pairs for its configuration.

Adding more options to the functionality simply requires the extension of the `PersistentVolumeClaimOptions` enum and adding the entries to the switch statement in `V1Controller.createPersistentVolumeClaims`.



Discussion / hints:
This is the command submitted to the analytic StatefulSet (acking example)...
```
'./heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://heronkube-zookeeper:2181/heron/acking-root-tag-0-7601280679230606520.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=acking51615bf4-a65b-4701-a9a3-ac65d8008fd6
          --topology-defn-file=acking.defn --state-manager-connection=heronkube-zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=heronkube --role=root --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
```
I believe the `SHARD_ID` is used internal to the pod to determine what to run. `SHARD_ID` seems to be created from the `POD_NAME` which is metadata.name in the StatefulSet.
```
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
```
In the Heron executor, this is the [logic](https://github.com/apache/incubator-heron/blob/ff7093f95c479cdd45278f1698c6a4782678219c/heron/executor/src/python/heron_executor.py#L990) that decides which processes are launched inside of the container. 
```
    if self.shard == 0:
      commands = self._get_tmanager_processes()
    else:
      self._untar_if_needed()
      commands = self._get_streaming_processes()
```
So the management StatefulSet should just work, but we will need to figure out how to increment the `SHARD_ID` by 1 for the executor processing StatefulSet. Maybe updating `SHARD_ID=${POD_NAME##*-}` to `SHARD_ID=$(expr ${POD_NAME##*-} + 1)` (or something better if your bash-fu is better than mine. :)
That last issue should be simple enough:

https://github.com/apache/incubator-heron/blob/4f588cfe003ee47264e564d25d694c1300d32fbc/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/V1Controller.java#L345-L347

becomes:

```java
  private static String setShardIdEnvironmentVariableCommand(boolean isExecutor) {
    final String pattern = String.format("%%s=%s && echo shardId=${%%s}",
        isExecutor ? "$((${POD_NAME##*-} + 1))" : "${POD_NAME##*-}");
    return String.format(pattern, ENV_SHARD_ID, ENV_SHARD_ID);
  }
```
We will need to create and teardown a `Service` to allow traffic between the `StatefulSet`s, I am thinking a single ingress/egress TCP/IP port of `6000` in the `executor` containers should suffice.
Yes, I think if we have a headless Service for each StatefulSet, they should be addressable. Heron will use Zookeeper for the discovery. Each process will self-register with their hostname which is provided by the headless Service. If you want, I can show you where in the Heron Executor that logic lives. It's basically a `hostname` type call.
That would be greatly appreciated, thank you.

Before work begins on this we need milestones to mark the completion of PRs. This will need to be multiple PRs.

**_PR1: Decouple the `manager` Pod._**
* Duplicate of the `executor` pods.
* Configure the `executor` and `manager` Pods using the dynamic `shard_id`. 
* Ability to set limits on resources for the `manager` using CLI arguments.
* Configure the `Service` to allow communication.

This is a big change and I will be limited in what I can programmatically test. The testing framework has limitations at its current version.

**_PR2: Pod Templates._**
* Individual Pod Templates for the `executor` and `manager`.

I have a plan in place to begin the first PR (Phase 1) once I know the PVC PR is stable.

* Executor containers will retain the name `executor`.
* Manager containers will have the name set to `manager`.
* Commands to set CPU limit: `heron.kubernetes.manager.limit.cpu` and RAM limit: `heron.kubernetes.manager.limit.memory`.
* `StatefulSet` for both will be duplicated.
* `executor` container boot argument will have `shard_id` in the range `[1 - n)`.
* `manager` container boot argument will have `shard_id` set to `0`.
* Create a `Service` with a single open TCP/IP port to facilitate communication. `Selector` set to match `Labels` for `topology` and `app`. Generate a re-creatable name using the `topology` name.
* `Selector` `matchLabels` for all toplogies `StatefulSet`s and `Service` set to `topology` and `app`.
* Remove both `StatefulSet`s using the [`deleteCollectionNamespacedStatefulSet`](https://javadoc.io/static/io.kubernetes/client-java-api/11.0.0/io/kubernetes/client/openapi/apis/AppsV1Api.html#deleteCollectionNamespacedStatefulSet-java.lang.String-java.lang.String-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-java.lang.String-java.lang.Integer-java.lang.Boolean-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-io.kubernetes.client.openapi.models.V1DeleteOptions-) with the `topology` and `apps` `Labels`.
* Remove the `Service` with [`deleteNamespacedService`](https://javadoc.io/static/io.kubernetes/client-java-api/11.0.0/io/kubernetes/client/openapi/apis/CoreV1Api.html#deleteNamespacedService-java.lang.String-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-java.lang.Boolean-java.lang.String-io.kubernetes.client.openapi.models.V1DeleteOptions-).

I think that should cover everything required to perform a basic decoupling of the `manager` and `executor` pods. We should also visit whether each `executor` container needs to have debugging ports to the tune of the number of Pods in the topology.

On the topic of creating a `Service`, I am wondering if we cannot just use the one which is already created for the topology:
https://github.com/apache/incubator-heron/blob/6b883bec716df6c58b86a5e3f12970ab4e72c1c6/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/V1Controller.java#L124-L131

https://github.com/apache/incubator-heron/blob/6b883bec716df6c58b86a5e3f12970ab4e72c1c6/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/V1Controller.java#L349-L369
@joshfischer1108 and @nicknezis preliminary PR has been created. I am running deployment testing on a Docker Ubuntu 18.04LTS container. There is a full battery of tests for the changes that I have made except for those changes which would require a testing base upgrade.
Deployment testing with `Persistent Volume Claim` created, a `Volume`, and `Volume Mounts` all created successfully. I have not configured a `StorageClass` which is why there is an error in the status of the Pod.

Manual cleanup of the `Persistent Volume Claim` is required and there will be a `conflict` error if the PVC names clash during submit.

<details><summary>Commands</summary>

```bash
~/bin/heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \
--verbose \
--config-property heron.kubernetes.pod.template.configmap.name=pod-templ-cf-map.pod-template.yaml \
org.apache.heron.examples.api.AckingTopology acking \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.claimName=volume-claim-name \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.storageClassName=storage-class-name \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=ReadWriteOnce,ReadOnlyMany \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=Block \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.path=path/to/mount \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.subPath=sub/path/to/mount
```

</details>

<details><summary>Persistent Volume Claim</summary>

```bash
Name:          volume-claim-name
Namespace:     default
StorageClass:  storage-class-name
Status:        Pending
Volume:        volumenameofchoice
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      0
Access Modes:  
VolumeMode:    Block
Used By:       acking-0
               acking-1
               acking-2
Events:        <none>

```

</details>

<details><summary>Describe Pod</summary>

```bash
Name:           acking-1
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=heron
                controller-revision-hash=acking-7fd8fb9fbd
                statefulset.kubernetes.io/pod-name=acking-1
                topology=acking
Annotations:    prometheus.io/port: 8080
                prometheus.io/scrape: true
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/acking
Containers:
  executor:
    Image:       apache/heron:testbuild
    Ports:       5555/TCP, 5556/UDP, 6001/TCP, 6002/TCP, 6003/TCP, 6004/TCP, 6005/TCP, 6006/TCP, 6007/TCP, 6008/TCP, 6009/TCP
    Host Ports:  0/TCP, 0/UDP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      sh
      -c
      ./heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--543424291072273987.tar.gz . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor --topology-name=acking --topology-id=acking32cfd6f1-a5f7-4472-a251-311397a61f3f --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/* --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*: --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824 --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/* --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003 --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007 --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009
    Limits:
      cpu:     3
      memory:  4Gi
    Requests:
      cpu:     3
      memory:  4Gi
    Environment:
      HOST:        (v1:status.podIP)
      POD_NAME:   acking-1 (v1:metadata.name)
      var_one:    variable one
      var_three:  variable three
      var_two:    variable two
    Mounts:
      /shared_volume from shared-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8scq5 (ro)
      path/to/mount from volumenameofchoice (rw,path="sub/path/to/mount")
  sidecar-container:
    Image:        alpine
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /shared_volume from shared-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8scq5 (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  shared-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  volumenameofchoice:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  volume-claim-name
    ReadOnly:   false
  kube-api-access-8scq5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 10s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 10s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  52s (x3 over 3m20s)  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
```

</details>

I'll hopefully get some time to test soon. But in the meantime I have a question.

Does this support mounting pre-existing PVCs and also dynamically creating the PVC? In the Spark implementation they had a special claim name `OnDemand` that would signify to create a dynamic PVC. With the current implementation, it seems we always create a new PVC. If so, this would preclude the ability to mount existing PVCs.

For the dynamic PVCs, I think it's ok that the PVC is not cleaned up dynamically when the analytic is killed. Perhaps that can be a config item on the scheduler that toggles the behavior of whether it should clean up any dynamically provisioned PVCs? We can always add this feature later. Not a blocker for merging this current Pull Request.
> Does this support mounting pre-existing PVCs and also dynamically creating the PVC? In the Spark implementation they had a special claim name `OnDemand` that would signify to create a dynamic PVC. With the current implementation, it seems we always create a new PVC. If so, this would preclude the ability to mount existing PVCs.

Correct, this implementation will always create a new dynamic PVC but the `OnDemand` functionality should be straightforward. There needs to be a CLI parameter for the `OnDemand` option and an addition to the `createPersistentVolumeClaims` routine to skip the PVC generation. The PVC to K8s deployment will then not deploy a PVC to the K8s cluster. The mounts and volumes will be added as they are now.

> For the dynamic PVCs, I think it's ok that the PVC is not cleaned up dynamically when the analytic is killed. Perhaps that can be a config item on the scheduler that toggles the behavior of whether it should clean up any dynamically provisioned PVCs? We can always add this feature later. Not a blocker for merging this current Pull Request.

I was looking into this and we need to understand the various components within Heron very well before making changes that will store resources created in the K8s cluster.

I am also working on trying to validate input for these configs (names and required parameters) because it is a hateful experience when the configs are bad. I do not think we should be doing any of this and that it should be the user's responsibility to get the commands right, what do you think? These checks add complexity and execution overhead during the submit process.
The reason Spark uses the `Claim Name` of `OnDemand` might be because it generates a PVC name based on the workload's name. If we were to do the same we could potentially get away with cleaning up on demand PVCs without having to update the scheduler. We would need a naming convention:

```bash
[Topology Name]-ondemand-[Volume-Name]
```

The question then becomes one of how we clean up all of the `OnDemand` claims. You could list the claims and generate a list of the ones with a prefix of `[Topology Name]-ondemand-` and axe them. We need to keep in mind that there may very well be thousands of claims and we cannot request a list by prefix from K8s.

We may be able to use the `fieldSelector` and `labelSelector` in [`listNamespacedPersistentVolumeClaim`](https://javadoc.io/static/io.kubernetes/client-java-api/11.0.0/io/kubernetes/client/openapi/apis/CoreV1Api.html#listNamespacedPersistentVolumeClaim-java.lang.String-java.lang.String-java.lang.Boolean-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-java.lang.String-java.lang.String-java.lang.Integer-java.lang.Boolean-) creatively to restrict returned values but we will need to add them to the PVC's on creation.
> The reason Spark uses the `Claim Name` of `OnDemand` might be because it generates a PVC name based on the workload's name. If we were to do the same we could potentially get away with cleaning up on demand PVCs without having to update the scheduler. We would need a naming convention:
> 
> ```shell
> [Topology Name]-ondemand-[Volume-Name]
> ```
> 

Correct. I think having the naming convention is fair decision because it is auto generated. If the scheduler is configured to cleanup dynamic PVCs, one way we can do this is with the naming convention. 

> The question then becomes one of how we clean up all of the `OnDemand` claims. You could list the claims and generate a list of the ones with a prefix of `[Topology Name]-ondemand-` and axe them. We need to keep in mind that there may very well be thousands of claims and we cannot request a list by prefix from K8s.
> 

Also we could add k8s labels to the PVCs which allows for a simple query to k8s to retrieve the list of PVCs that match the Selector that we would create.

This is an [old version](https://github.com/apache/incubator-heron/blob/fd7fde0518562d60a7a85679715e2b15ee464d37/heron/schedulers/src/java/com/twitter/heron/scheduler/kubernetes/KubernetesCompat.java#L37) of the Heron k8s scheduler that used to delete Pods (before using StatefulSet). Note how we create a Selector that is used in the subsequent delete call. The Selector matched the labels which were on the PodSpec [here](https://github.com/apache/incubator-heron/blob/fd7fde0518562d60a7a85679715e2b15ee464d37/heron/schedulers/src/java/com/twitter/heron/scheduler/kubernetes/AppsV1beta1Controller.java#L286).

So if we are making the PVCs, we can add a combination of labels that will be specific enough to not auto-delete other PVCs. Then use these labels to know what to delete.

> We may be able to use the `fieldSelector` and `labelSelector` in [`listNamespacedPersistentVolumeClaim`](https://javadoc.io/static/io.kubernetes/client-java-api/11.0.0/io/kubernetes/client/openapi/apis/CoreV1Api.html#listNamespacedPersistentVolumeClaim-java.lang.String-java.lang.String-java.lang.Boolean-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-java.lang.String-java.lang.String-java.lang.Integer-java.lang.Boolean-) creatively to restrict returned values but we will need to add them to the PVC's on creation.

Yup!
Thank you @nicknezis that is very helpful info!

I think we can add two selector labels to the PVC:

```yaml
selector:
  topology: <topology-name>
  onDemand: true
```

Then we can form the `labelSelector` as:

```java
topology=<topology-name>,onDemand=true
```

I think this should be minimally specific enough to act as a candidate key to select all PVC's associated with a specific topology.

As for the `Claim Name` I think we need something unique enough to not cause collisions in the cluster:

```java
ondemand-[Topology Name]-[Volume-Name]
```
Most of the last piece of the puzzle for supporting both dynamic and statically backed PVCs is now complete. The final piece is the clean up of the dynamically backed/provisioned PVCs which I am debugging right now in Minikube.

@nicknezis I am hoping you have some experience with this having transitioned Heron to `StatefulSet`s. I have the `Selector matchLabels` set up correctly, as indicated by the PVC's YAML below, but the `matchLabels` are unable to select the PVC. I am employing the [`deleteCollectionNamespacedPersistentVolumeClaim`](https://javadoc.io/doc/io.kubernetes/client-java-api/11.0.0/io/kubernetes/client/openapi/apis/CoreV1Api.html#deleteCollectionNamespacedPersistentVolumeClaim-java.lang.String-java.lang.String-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-java.lang.String-java.lang.Integer-java.lang.Boolean-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-io.kubernetes.client.openapi.models.V1DeleteOptions-) method. I can confirm that the `matchLabel` pattern being generated is as follows:

```yaml
onDemand=true,topology=acking
```

<details><summary>PVC Spec</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2021-11-07T16:27:51Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: ondemand-acking-volumenameofchoice
  namespace: default
  resourceVersion: "714"
  uid: 38f7a7fb-12f2-4315-a5bd-4f346061fdc9
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  selector:
    matchLabels:
      onDemand: "true"
      topology: acking
  storageClassName: storage-class-name
  volumeMode: Block
  volumeName: volumenameofchoice
status:
  phase: Pending
```

</details>

I have also attempted to match against a single `matchLabel` of `topology` to no avail.

<details><summary>Submit Command</summary>

```yaml
~/bin/heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \
--verbose \
--config-property heron.kubernetes.pod.template.configmap.name=pod-templ-cf-map.pod-template.yaml \
org.apache.heron.examples.api.AckingTopology acking \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.claimName=OnDemand \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.storageClassName=storage-class-name \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=ReadWriteOnce,ReadOnlyMany \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=Block \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.path=path/to/mount \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.subPath=sub/path/to/mount
```

</details>

I  am also testing by loading the PVC spec into K8s and using Kubectl to select the PVC with issues:

```bash
❯ minikube kubectl -- get persistentvolumeclaims --selector=topology=acking
No resources found in default namespace.
```

***Edit:*** I did some reading and I think I have this figured out. I confused the purpose of the `matchLabel`.
So I think the issue may be that you are using the Selector Match Labels when really you need to put Labels in the Object Metadata. Take a look at this [old code](https://github.com/apache/incubator-heron/blob/fd7fde0518562d60a7a85679715e2b15ee464d37/heron/schedulers/src/java/com/twitter/heron/scheduler/kubernetes/AppsV1beta1Controller.java#L296) that put labels on the PodSpec.

The PVC's Selector (as is the case for other objects also) is used as the query to match on something else's metadata labels. In this case, a PVC can be configured with a Selector to find specific types of Persistent Volumes (PV).

Edit: Just noticed your edit. LOL. Sounds like you quickly realized the difference of selector vs metadata labels. I'll leave my comment above in case it's helpful for anyone else reading this thread.
Thank you for looking into this and confirming my suspicion, I genuinely appreciate having someone else looking over things as well :smile:. I have confused `matchLabels` with `labels` that are matched to rather than ones that will be matched to other resources.

I have run the full battery of tests locally as well as completed deployment testing. I believe the production code is stable enough for broader testing. I will be reviewing the test suite and expanding it if necessary.

The only addition I feel I can make to the production code is to validate input for lowercase `RFC-1123` compliance. This will slow the code down because of regex matching. Other than this, I believe we have a feature RC here.
I have added lowercase `RFC-1123` validation for `Volume`, `Claim`, as well as `Storage Class` names. I have tested the validation through the test suite but have not had the opportunity to run deployment testing with them. The full battery of tests is passing locally. This should hopefully catch errors before anything gets handed over to K8s.
Just started some testing in Kubernetes. Deployed the `acking` topology, but when I tried to kill it, I received the following: `Failed to connect to K8s cluster to delete Persistent Volume Claims for topology `acking`. A manual clean-up is required.`

This was a submission without enabling any PVCs for the pods.

**_Edit:_** Disregard, I think I used an old Helm chart that didn't have the new PVC permissions.
No worries, I am not sure if the Helm Chart I edited is minimal enough as far as permissions go - can we get away without the `list` permission? I am performing all of my testing on Minikube in a clean environment (`delete --all`) and perform a manual deployment each time.
So even after adding the permissions, I still get the forbidden failure when deleting. I'll keep investigating.

I did continue testing with the `ondemand`. It was able to create the PVC, but they stay in a pending state. I'm not sure, but might be an issue with how I configured things. One potential issue or design consideration... it seems to only make a single PVC for the entire topology. I was assuming it would be a PVC per pod. Here is the naming convention from the [Spark code](https://github.com/apache/spark/blob/de59e01aa4853ef951da080c0d1908d53d133ebe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala#L76). It seems they have an index `$i` on the end as they loop through?
Could you please try to dump the PVC with the `get persistentvolumeclaims [claim-name] -o yaml` and see if something is off? You can do the same for the Pod's configs.

I am able to `create` and `delete` all the generated PVC's in Minikube, so I think this is a permission-based issue. I to see the status as in `pending` but that is because I either do not have the claims setup for statically backed storage or the dynamically backed storage enabled and configured.
> One potential issue or design consideration... it seems to only make a single PVC for the entire topology. I was assuming it would be a PVC per pod. Here is the naming convention from the [Spark code](https://github.com/apache/spark/blob/de59e01aa4853ef951da080c0d1908d53d133ebe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala#L76). It seems they have an index `$i` on the end as they loop through?

The design is such that a single PVC is generated per dynamically allocated/backed volume for the entire `StatefulSet`. All of the `executor` pods will gain entries added for the `Volume Mounts` and the entire pod template will receive an entry for the shared `Volume`. Since it is a shared `Volume` for the `StatefulSet`, my assumption being this is to support amongst many other things, a shared disk between support containers makes perfect sense. You may allocate as many PVCs as you want for topology, though.

**_Edit:_** If we can use variables in the K8s Pod Template which K8s can resolve, specifically for the Pod number, then this can be done. Say we have the variable `$pod.number` then I would only need to update the following two entries in the configs:
`PersistentVolumeClaim.metadata.name`: ondemand-[topology-name]-[volume-name]-$pod.number
`Volumes.name`: [volume-name]-$pod.number

With this, we make the above field entries with variables in the Pod Templates. When the PVCs are deployed to K8s we add the `$pod.number` suffix to the PVC names on the fly. This is all dependant on the ability to extract the Pod number via a variable.

*Nevertheless, if there is no way we can reference fields in the Pod Template in other fields within the template, I can not see how this can be achieved. There would need to be API access to the individual Pods within the `StatefulSet` to adjust the fields for `Volume Mounts`. Creating the `Volume` and `Persistent Volume Claims` remains a nonissue.*
I can confirm the ability to launch and kill a topology with/without a Pod Template as well as being able to launch a topology with a statically and dynamically backed PVC. The dynamically backed PVC is deleted on kill. I am testing in Minikube on a clean environment (`delete --all`) with the latest PR build.

<details><summary>Without dynamic PVC: kubectl get pod acking-1 -o yaml</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-10T22:36:19Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-7cf596b959
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 7638b318-645b-41e1-b0c9-47e124207a13
  resourceVersion: "852"
  uid: 1e125904-6c32-4c69-a48a-ba85a4136d9b
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--358503390991778356.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=acking8c323447-bce6-4a35-9b2f-7b6e1b9cc46d
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-lbslq
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-lbslq
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - emptyDir: {}
    name: shared-volume
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volume-claim-name
  - name: kube-api-access-lbslq
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-10T22:36:19Z"
    message: '0/1 nodes are available: 1 persistentvolumeclaim "volume-claim-name"
      not found.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>With dynamic PVC: kubectl get pod acking-1 -o yaml</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-10T22:39:57Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-59fdd8c858
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: dc5ae063-672f-4b1c-98cd-0936cd02a1b0
  resourceVersion: "1060"
  uid: cbe65fc2-cddf-496d-b633-68fad946c9b1
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--4583400108540459301.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=acking1f62dd17-33f9-4a72-98e7-daca680e9f16
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-njblm
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-njblm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - emptyDir: {}
    name: shared-volume
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: ondemand-acking-volumenameofchoice
  - name: kube-api-access-njblm
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-10T22:39:57Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>kubectl get persistentvolumeclaims ondemand-acking-volumenameofchoice -o yaml</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2021-11-10T22:39:57Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    onDemand: "true"
    topology: acking
  name: ondemand-acking-volumenameofchoice
  namespace: default
  resourceVersion: "1048"
  uid: 34a9bb66-db51-4abe-bc65-a8b45eef1593
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: storage-class-name
  volumeMode: Block
  volumeName: volumenameofchoice
status:
  phase: Pending
```

</details>

One thought for the `ondemand` approach would be that perhaps we should be leveraging the `VolumeClaimTemplate` feature of a StatefulSet. Here is an [example](https://stackoverflow.com/questions/53927587/kubernetes-multiple-pvc-with-statefulset-for-each-pod-vs-single-pvc-for-all-pods). This would allow for a PVC per pod without having to manually create them. Kubernetes will automatically create with the pod's ordinal number. This might also help answer how we handle the `addContainer` call which can be called when someone rescales a running Heron topology. In this case any new pods would get a new PVC. I think it's ok to leave the PVC if the analytic is scaled down because we will cleanup all PVCs when the topology is killed.

**_Edit:_** This would also allow us to keep most of your logic the same. I do agree there is benefit in having the single shared volume so don't want to lose that. I perhaps was describing things with too strong a focus on the `ondemand` use case.
`VolumeClaimTemplate`s is a cleaner way of handling all this, I shall switch over to using them. I think we can remove the shared PVC volume; there are certain things that should be left to being manually created in Pod Templates. I need to figure out how to rejig the code.

On a side note, we need to fix the CI issues. The `statemgr_unit` test needs to be fixed and we _need_ the ability to manually retrigger the CI pipeline.

**_Edit:_** switching to the `VolumeClaimTemplate` seems to be ~a simple~ an "involved" adjustment with a lot of test updates. I am figuring things out.

We are going to need to drop the `claimName` option and switch to only using only `Volume Claim Templates` going forward. This will allow people to set up a statically backed PV or expect a dynamically backed one using the `storageClassName` option. If nothing is provided for the `storageClassName` a statically backed PV will be expected. This means PVCs will always be generated and deployed whenever anyone uses these CLI options. Thoughts?
@nicknezis: Setup and tear down of all PVCs with/without Pod Templates as well as testing without PVCs successful. I have dumped the `YAML` below to make life easier. Full testing battery passing locally.

**_STATIC:_**

<details><summary>StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-12T01:56:24Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "1421"
  uid: 6e991292-575c-4cbb-b200-3e4a315cf4b6
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--5861445565125076364.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=ackingac5a762c-c2d2-45fc-8b59-f83acfff7564
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        onDemand: "true"
        topology: acking
      name: volumenameofchoice
    spec:
      accessModes:
      - ReadWriteOnce
      - ReadOnlyMany
      resources:
        requests:
          storage: 555Gi
      volumeMode: Block
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-8688f48fb8
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-8688f48fb8
  updatedReplicas: 3
```

</details>

<details><summary>Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-12T01:50:55Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-5f56945486
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 22b719a9-9b7d-4d63-8365-e76b44a50abe
  resourceVersion: "960"
  uid: 35118297-0949-4648-b12f-8b755e583436
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--1817385379878734459.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=ackingff8ff63d-977f-49ba-8521-11375f9c41d6
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-b8b88
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-b8b88
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volumenameofchoice-acking-1
  - emptyDir: {}
    name: shared-volume
  - name: kube-api-access-b8b88
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-12T01:50:55Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>PVC</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
  creationTimestamp: "2021-11-12T01:50:55Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: heron
    onDemand: "true"
    topology: acking
  name: volumenameofchoice-acking-0
  namespace: default
  resourceVersion: "954"
  uid: 0e8c8aec-ce54-410e-8790-5da2238af798
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: standard
  volumeMode: Block
status:
  phase: Pending
```

</details>

**_DYNAMIC:_**

<details><summary>StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-12T01:53:56Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "1212"
  uid: 597877ea-4601-4bf0-80e3-72f9db78ad47
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--6172080214638514375.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=acking6a21ddc3-dbab-4c25-8a54-dcd3b0264b55
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        onDemand: "true"
        topology: acking
      name: volumenameofchoice
    spec:
      accessModes:
      - ReadWriteOnce
      - ReadOnlyMany
      resources:
        requests:
          storage: 555Gi
      storageClassName: storage-class-name
      volumeMode: Block
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-5cbfb4df4c
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-5cbfb4df4c
  updatedReplicas: 3
```

</details>

<details><summary>Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-12T01:53:56Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-5cbfb4df4c
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 597877ea-4601-4bf0-80e3-72f9db78ad47
  resourceVersion: "1205"
  uid: eec86082-2259-4fe7-999a-10f7e58d2e79
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--6172080214638514375.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=acking6a21ddc3-dbab-4c25-8a54-dcd3b0264b55
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-bdwcl
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-bdwcl
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volumenameofchoice-acking-1
  - emptyDir: {}
    name: shared-volume
  - name: kube-api-access-bdwcl
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-12T01:53:56Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>PVC</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2021-11-12T01:53:56Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: heron
    onDemand: "true"
    topology: acking
  name: volumenameofchoice-acking-0
  namespace: default
  resourceVersion: "1190"
  uid: b7a4c403-b749-452b-ac70-16c6a9e158d6
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: storage-class-name
  volumeMode: Block
status:
  phase: Pending
```

</details>
I have updated the usage [documentation](https://github.com/apache/incubator-heron/blob/274f86c3567cd8633eb8c97e968e0c35d06e903c/website2/docs/schedulers-k8s-persistent-volume-claims.md) and merged in the updates. Let us see if TravisCI is in a good mood 🎲 🤞🏼...

**_Edit:_** Updated with potential use cases  [documentation](https://github.com/apache/incubator-heron/blob/c7a97694e111fa3e4bf222fe13ee90d9638c3786/website2/docs/schedulers-k8s-persistent-volume-claims.md).
I figured out why I couldn't delete any topologies. Our Helm chart didn't have the `deletecollection` permission. Added it to the branch.
Apologies for the delay in testing. Now that the other Python issue is resolved, I'm again able to submit and kill topologies. It seems that the scheduler is always attempting to create new PVCs. 

I created the following PVC:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nez-big-storage
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: standard
  volumeMode: Filesystem
```

And then submitted the following: 
```
heron submit heronkube \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.nez-volume-name.readOnly=true \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.nez-volume-name.path=/tmp/nez \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.nez-volume-name.subPath=was/here \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.nez-volume-name.claimname=nez-big-storage \
/heron/examples/heron-api-examples.jar org.apache.heron.examples.api.AckingTopology acking
```

The pre-provisioned PVC would be mounted at the above path `/tmp/nez` with a `volumeMount` named `nez-volume-name`. Not this is the volume name, as opposed to the volume claim name.
The "dynamic" creation of the PVCs would only happen when the special claim name `OnDemand` is used. This would be a special toggle that triggers the logic to dynamically create the PVC with the Claim Template. Only then do we use the other options to populate the Claim Template.

I think we're getting close. Hopefully that explanation helps.

One thing I notice on the Spark config structure that might be worth noting, they use the `mount` and `options` to distinguish the volume mount set of configs and the options that are specific to the type of storage. In our case, so far we are only implementing the PVC approach, but in the future we might also add `hostPath`. I'm not sure if having the `mount` and `options` configuration hierarchy helps make the code cleaner. I can see things still working the way you currently have the naming, so it might not be a big deal.
Thank you for taking the time to look at this!

I think it is important to keep in mind the following key limitations of a `StatefulSet`:

* Pod templates are copied verbatim.
* Volumes are shared with all Pods.
* The same volume mounts are duplicated in matching containers across Pods.
* Only way to have unique volumes per pod is to leverage `volumeClaimTemplates`.

Please see [***Stable Storage***](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/).

If you would like to mount a volume you have already created you need to name it based on the naming convention K8s uses. K8s will initially attempt locating a persistent volume that matches the persistent volume claim, if that fails it attempts to generate a dynamically backed volume.

If I proceed to add volumes to the `StatefulSet`, those volumes will be shared in all Pods. To mount a volume in a Pod you need to have a linked Volume in the Pod's spec. We could have some bash trickery that mounts the unique Volume into a container, but that volume will be shared with all other Pods. The long and the short of it is that we will end up right back where we started with the Volume being shared with all Pods.

It is bad design to utilize the same CLI parameter to mount both unique and shared volumes.

Yes, the case where we mount an existing PVC, this would be a shared PVC accessed by all pods. (i.e. `ReadWriteMany` or `ReadOnlyMany`)

One `ReadOnlyMany` example is you could mount a read only PVC with files such as certificates and certificate authorities. 

For `ReadWriteMany` you can also mount a shared drive for large file access. There are cases in which streaming data is smaller references to larger files. The bolts would have access to the shared volumes to access larger files. The bolts could do file scan actions for specific files based on the metadata only stream.

The case for the dynamic per-pod mounts using the PersistentVolumeClaim Template would be for local checkpointing and log file storage. Currently in each pod there is a `log-files` folder in each container that writes the Spout/Bolt logs to the container's built in storage. With the dynamic PVC mount, one could configure to use a system like Rook/Ceph to provide dynamic storage that does not use the Docker volume which might be more limited.

If there is a better suggestion for the CLI parameters, I'm cool with that. I was merely using the Spark design as a starting point. I see the value in how they structured the subsets of config under `mount` and `options` that supports handling the different storage types (i.e. `nfs`, `pvc`, `hostPath`, etc). But we could totally have a more explicit set of config.

I think it's my fault for discussing both the static mount and dynamic mont. Hopefully I didn't confuse things too much. I really was looking to mirror what Spark provided in terms of the functionality. I think their `OnDemand` and `local` are the only cases which get the per pod specific storage. The other cases, the pods share the same storage.
I am going to go ahead and reintroduce the mounts and volumes for shared PVCs at which point I shall consider this PR completed. As careful as I have been, and as extensive as the test suite is, 1000+ lines of code introduce a lot of moving parts that could go wrong.

> If there is a better suggestion for the CLI parameters, I'm cool with that.

Individual volumes: `heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.`
Shared volumes: `heron.kubernetes.volumes.persistentVolumeClaim.shared.volumeNameOfChoice.`

But that is an issue for another time.
I added support for shared `Volume`s via a `claimName` but have not had the chance to complete a full battery of deployment tests.

A Claim name of `OnDemand` will result in PVCs being created. Any other Claim name will only add `Volume`s to the Pod Spec and `Volume Mounts` to the `Executor` Pod and will be replicated between all Pods in the topology.

Documentation is [here](https://github.com/apache/incubator-heron/blob/6a449b178f892f1bd109ba28ead228abc5a52613/website2/docs/schedulers-k8s-persistent-volume-claims.md).

I think we may be able to whittle down the permissions for the PVC to:

```yaml
rules:
- apiGroups: 
 - ""
 resources: 
 - persistentvolumeclaims
 verbs: 
 - create
 - deletecollection
```
I ran some quick tests before I begin separating the `executor`s and the `manager` and everything looks good. `YAML` for each case is below, I can confirm that a shared volume will not add a PVC.

<details><summary>Shared StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-18T17:23:24Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "2717"
  uid: f4e9b2a9-f880-4b80-bd15-7cd54cdd23fb
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0-160636999515090835.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=ackingcf66189f-ca7f-46a3-a0a9-cb41a5d658c4
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
      - name: volumenameofchoice
        persistentVolumeClaim:
          claimName: requested-claim-by-user
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-76494fb95
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-76494fb95
  updatedReplicas: 3
```

</details>

<details><summary>Shared Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-18T17:23:24Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-76494fb95
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: f4e9b2a9-f880-4b80-bd15-7cd54cdd23fb
  resourceVersion: "2718"
  uid: 0e28c20c-e3aa-4f36-a2dd-da063147f5c0
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0-160636999515090835.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=ackingcf66189f-ca7f-46a3-a0a9-cb41a5d658c4
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-8pmpr
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-8pmpr
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - emptyDir: {}
    name: shared-volume
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: requested-claim-by-user
  - name: kube-api-access-8pmpr
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-18T17:23:24Z"
    message: '0/1 nodes are available: 1 persistentvolumeclaim "requested-claim-by-user"
      not found.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>Static StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-18T17:12:52Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "2061"
  uid: 54337851-2bc2-4cf3-bf66-d9e8a5716aa2
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--1619810271492859618.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=acking87b0a985-49ec-4749-aa2d-d7f4aaa736a4
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        onDemand: "true"
        topology: acking
      name: volumenameofchoice
    spec:
      accessModes:
      - ReadWriteOnce
      - ReadOnlyMany
      resources:
        requests:
          storage: 555Gi
      volumeMode: Block
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-6c4c987ddc
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-6c4c987ddc
  updatedReplicas: 3
```

</details>

<details><summary>Static Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-18T17:12:52Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-6c4c987ddc
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 54337851-2bc2-4cf3-bf66-d9e8a5716aa2
  resourceVersion: "2051"
  uid: 027fff42-3f1e-4457-a3f8-6505b256f3af
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--1619810271492859618.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=acking87b0a985-49ec-4749-aa2d-d7f4aaa736a4
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-txlbw
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-txlbw
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volumenameofchoice-acking-1
  - emptyDir: {}
    name: shared-volume
  - name: kube-api-access-txlbw
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-18T17:12:52Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>Static PVC</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
  creationTimestamp: "2021-11-18T17:12:52Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: heron
    onDemand: "true"
    topology: acking
  name: volumenameofchoice-acking-1
  namespace: default
  resourceVersion: "2047"
  uid: 50a226ac-d7cb-44b9-8d55-8ecc9bf8796d
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: standard
  volumeMode: Block
status:
  phase: Pending
```

</details>

<details><summary>Dynamic StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-18T17:17:59Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "2412"
  uid: 4263ad61-4d70-4803-9664-5e1cc04c174e
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--8727299585809965289.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=ackinge77c4afd-1609-4616-8344-a3474f0cd2ba
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        onDemand: "true"
        topology: acking
      name: volumenameofchoice
    spec:
      accessModes:
      - ReadWriteOnce
      - ReadOnlyMany
      resources:
        requests:
          storage: 555Gi
      storageClassName: storage-class-name
      volumeMode: Block
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-7bfcb94659
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-7bfcb94659
  updatedReplicas: 3
```

</details>

<details><summary>Dynamic Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-18T17:17:59Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-7bfcb94659
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 4263ad61-4d70-4803-9664-5e1cc04c174e
  resourceVersion: "2406"
  uid: 807570fd-1a38-4a5a-bb5e-cb5fe13acecc
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--8727299585809965289.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=ackinge77c4afd-1609-4616-8344-a3474f0cd2ba
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wl7h4
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wl7h4
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volumenameofchoice-acking-1
  - emptyDir: {}
    name: shared-volume
  - name: kube-api-access-wl7h4
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-18T17:17:59Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>Dynamic PVC</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2021-11-18T17:17:59Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: heron
    onDemand: "true"
    topology: acking
  name: volumenameofchoice-acking-1
  namespace: default
  resourceVersion: "2395"
  uid: 65a9291e-ea83-41bb-a1ce-56d76574f0f3
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: storage-class-name
  volumeMode: Block
status:
  phase: Pending
```

</details>
@joshfischer1108 and @nicknezis preliminary PR has been created. I am running deployment testing on a Docker Ubuntu 18.04LTS container. There is a full battery of tests for the changes that I have made except for those changes which would require a testing base upgrade.
Deployment testing with `Persistent Volume Claim` created, a `Volume`, and `Volume Mounts` all created successfully. I have not configured a `StorageClass` which is why there is an error in the status of the Pod.

Manual cleanup of the `Persistent Volume Claim` is required and there will be a `conflict` error if the PVC names clash during submit.

<details><summary>Commands</summary>

```bash
~/bin/heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \
--verbose \
--config-property heron.kubernetes.pod.template.configmap.name=pod-templ-cf-map.pod-template.yaml \
org.apache.heron.examples.api.AckingTopology acking \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.claimName=volume-claim-name \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.storageClassName=storage-class-name \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=ReadWriteOnce,ReadOnlyMany \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=Block \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.path=path/to/mount \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.subPath=sub/path/to/mount
```

</details>

<details><summary>Persistent Volume Claim</summary>

```bash
Name:          volume-claim-name
Namespace:     default
StorageClass:  storage-class-name
Status:        Pending
Volume:        volumenameofchoice
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      0
Access Modes:  
VolumeMode:    Block
Used By:       acking-0
               acking-1
               acking-2
Events:        <none>

```

</details>

<details><summary>Describe Pod</summary>

```bash
Name:           acking-1
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=heron
                controller-revision-hash=acking-7fd8fb9fbd
                statefulset.kubernetes.io/pod-name=acking-1
                topology=acking
Annotations:    prometheus.io/port: 8080
                prometheus.io/scrape: true
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/acking
Containers:
  executor:
    Image:       apache/heron:testbuild
    Ports:       5555/TCP, 5556/UDP, 6001/TCP, 6002/TCP, 6003/TCP, 6004/TCP, 6005/TCP, 6006/TCP, 6007/TCP, 6008/TCP, 6009/TCP
    Host Ports:  0/TCP, 0/UDP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      sh
      -c
      ./heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--543424291072273987.tar.gz . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor --topology-name=acking --topology-id=acking32cfd6f1-a5f7-4472-a251-311397a61f3f --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/* --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*: --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824 --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/* --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003 --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007 --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009
    Limits:
      cpu:     3
      memory:  4Gi
    Requests:
      cpu:     3
      memory:  4Gi
    Environment:
      HOST:        (v1:status.podIP)
      POD_NAME:   acking-1 (v1:metadata.name)
      var_one:    variable one
      var_three:  variable three
      var_two:    variable two
    Mounts:
      /shared_volume from shared-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8scq5 (ro)
      path/to/mount from volumenameofchoice (rw,path="sub/path/to/mount")
  sidecar-container:
    Image:        alpine
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /shared_volume from shared-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8scq5 (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  shared-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  volumenameofchoice:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  volume-claim-name
    ReadOnly:   false
  kube-api-access-8scq5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 10s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 10s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  52s (x3 over 3m20s)  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
```

</details>

I'll hopefully get some time to test soon. But in the meantime I have a question.

Does this support mounting pre-existing PVCs and also dynamically creating the PVC? In the Spark implementation they had a special claim name `OnDemand` that would signify to create a dynamic PVC. With the current implementation, it seems we always create a new PVC. If so, this would preclude the ability to mount existing PVCs.

For the dynamic PVCs, I think it's ok that the PVC is not cleaned up dynamically when the analytic is killed. Perhaps that can be a config item on the scheduler that toggles the behavior of whether it should clean up any dynamically provisioned PVCs? We can always add this feature later. Not a blocker for merging this current Pull Request.
> Does this support mounting pre-existing PVCs and also dynamically creating the PVC? In the Spark implementation they had a special claim name `OnDemand` that would signify to create a dynamic PVC. With the current implementation, it seems we always create a new PVC. If so, this would preclude the ability to mount existing PVCs.

Correct, this implementation will always create a new dynamic PVC but the `OnDemand` functionality should be straightforward. There needs to be a CLI parameter for the `OnDemand` option and an addition to the `createPersistentVolumeClaims` routine to skip the PVC generation. The PVC to K8s deployment will then not deploy a PVC to the K8s cluster. The mounts and volumes will be added as they are now.

> For the dynamic PVCs, I think it's ok that the PVC is not cleaned up dynamically when the analytic is killed. Perhaps that can be a config item on the scheduler that toggles the behavior of whether it should clean up any dynamically provisioned PVCs? We can always add this feature later. Not a blocker for merging this current Pull Request.

I was looking into this and we need to understand the various components within Heron very well before making changes that will store resources created in the K8s cluster.

I am also working on trying to validate input for these configs (names and required parameters) because it is a hateful experience when the configs are bad. I do not think we should be doing any of this and that it should be the user's responsibility to get the commands right, what do you think? These checks add complexity and execution overhead during the submit process.
The reason Spark uses the `Claim Name` of `OnDemand` might be because it generates a PVC name based on the workload's name. If we were to do the same we could potentially get away with cleaning up on demand PVCs without having to update the scheduler. We would need a naming convention:

```bash
[Topology Name]-ondemand-[Volume-Name]
```

The question then becomes one of how we clean up all of the `OnDemand` claims. You could list the claims and generate a list of the ones with a prefix of `[Topology Name]-ondemand-` and axe them. We need to keep in mind that there may very well be thousands of claims and we cannot request a list by prefix from K8s.

We may be able to use the `fieldSelector` and `labelSelector` in [`listNamespacedPersistentVolumeClaim`](https://javadoc.io/static/io.kubernetes/client-java-api/11.0.0/io/kubernetes/client/openapi/apis/CoreV1Api.html#listNamespacedPersistentVolumeClaim-java.lang.String-java.lang.String-java.lang.Boolean-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-java.lang.String-java.lang.String-java.lang.Integer-java.lang.Boolean-) creatively to restrict returned values but we will need to add them to the PVC's on creation.
> The reason Spark uses the `Claim Name` of `OnDemand` might be because it generates a PVC name based on the workload's name. If we were to do the same we could potentially get away with cleaning up on demand PVCs without having to update the scheduler. We would need a naming convention:
> 
> ```shell
> [Topology Name]-ondemand-[Volume-Name]
> ```
> 

Correct. I think having the naming convention is fair decision because it is auto generated. If the scheduler is configured to cleanup dynamic PVCs, one way we can do this is with the naming convention. 

> The question then becomes one of how we clean up all of the `OnDemand` claims. You could list the claims and generate a list of the ones with a prefix of `[Topology Name]-ondemand-` and axe them. We need to keep in mind that there may very well be thousands of claims and we cannot request a list by prefix from K8s.
> 

Also we could add k8s labels to the PVCs which allows for a simple query to k8s to retrieve the list of PVCs that match the Selector that we would create.

This is an [old version](https://github.com/apache/incubator-heron/blob/fd7fde0518562d60a7a85679715e2b15ee464d37/heron/schedulers/src/java/com/twitter/heron/scheduler/kubernetes/KubernetesCompat.java#L37) of the Heron k8s scheduler that used to delete Pods (before using StatefulSet). Note how we create a Selector that is used in the subsequent delete call. The Selector matched the labels which were on the PodSpec [here](https://github.com/apache/incubator-heron/blob/fd7fde0518562d60a7a85679715e2b15ee464d37/heron/schedulers/src/java/com/twitter/heron/scheduler/kubernetes/AppsV1beta1Controller.java#L286).

So if we are making the PVCs, we can add a combination of labels that will be specific enough to not auto-delete other PVCs. Then use these labels to know what to delete.

> We may be able to use the `fieldSelector` and `labelSelector` in [`listNamespacedPersistentVolumeClaim`](https://javadoc.io/static/io.kubernetes/client-java-api/11.0.0/io/kubernetes/client/openapi/apis/CoreV1Api.html#listNamespacedPersistentVolumeClaim-java.lang.String-java.lang.String-java.lang.Boolean-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-java.lang.String-java.lang.String-java.lang.Integer-java.lang.Boolean-) creatively to restrict returned values but we will need to add them to the PVC's on creation.

Yup!
Thank you @nicknezis that is very helpful info!

I think we can add two selector labels to the PVC:

```yaml
selector:
  topology: <topology-name>
  onDemand: true
```

Then we can form the `labelSelector` as:

```java
topology=<topology-name>,onDemand=true
```

I think this should be minimally specific enough to act as a candidate key to select all PVC's associated with a specific topology.

As for the `Claim Name` I think we need something unique enough to not cause collisions in the cluster:

```java
ondemand-[Topology Name]-[Volume-Name]
```
Most of the last piece of the puzzle for supporting both dynamic and statically backed PVCs is now complete. The final piece is the clean up of the dynamically backed/provisioned PVCs which I am debugging right now in Minikube.

@nicknezis I am hoping you have some experience with this having transitioned Heron to `StatefulSet`s. I have the `Selector matchLabels` set up correctly, as indicated by the PVC's YAML below, but the `matchLabels` are unable to select the PVC. I am employing the [`deleteCollectionNamespacedPersistentVolumeClaim`](https://javadoc.io/doc/io.kubernetes/client-java-api/11.0.0/io/kubernetes/client/openapi/apis/CoreV1Api.html#deleteCollectionNamespacedPersistentVolumeClaim-java.lang.String-java.lang.String-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-java.lang.String-java.lang.Integer-java.lang.Boolean-java.lang.String-java.lang.String-java.lang.String-java.lang.Integer-io.kubernetes.client.openapi.models.V1DeleteOptions-) method. I can confirm that the `matchLabel` pattern being generated is as follows:

```yaml
onDemand=true,topology=acking
```

<details><summary>PVC Spec</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2021-11-07T16:27:51Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: ondemand-acking-volumenameofchoice
  namespace: default
  resourceVersion: "714"
  uid: 38f7a7fb-12f2-4315-a5bd-4f346061fdc9
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  selector:
    matchLabels:
      onDemand: "true"
      topology: acking
  storageClassName: storage-class-name
  volumeMode: Block
  volumeName: volumenameofchoice
status:
  phase: Pending
```

</details>

I have also attempted to match against a single `matchLabel` of `topology` to no avail.

<details><summary>Submit Command</summary>

```yaml
~/bin/heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \
--verbose \
--config-property heron.kubernetes.pod.template.configmap.name=pod-templ-cf-map.pod-template.yaml \
org.apache.heron.examples.api.AckingTopology acking \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.claimName=OnDemand \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.storageClassName=storage-class-name \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=ReadWriteOnce,ReadOnlyMany \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=Block \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.path=path/to/mount \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.subPath=sub/path/to/mount
```

</details>

I  am also testing by loading the PVC spec into K8s and using Kubectl to select the PVC with issues:

```bash
❯ minikube kubectl -- get persistentvolumeclaims --selector=topology=acking
No resources found in default namespace.
```

***Edit:*** I did some reading and I think I have this figured out. I confused the purpose of the `matchLabel`.
So I think the issue may be that you are using the Selector Match Labels when really you need to put Labels in the Object Metadata. Take a look at this [old code](https://github.com/apache/incubator-heron/blob/fd7fde0518562d60a7a85679715e2b15ee464d37/heron/schedulers/src/java/com/twitter/heron/scheduler/kubernetes/AppsV1beta1Controller.java#L296) that put labels on the PodSpec.

The PVC's Selector (as is the case for other objects also) is used as the query to match on something else's metadata labels. In this case, a PVC can be configured with a Selector to find specific types of Persistent Volumes (PV).

Edit: Just noticed your edit. LOL. Sounds like you quickly realized the difference of selector vs metadata labels. I'll leave my comment above in case it's helpful for anyone else reading this thread.
Thank you for looking into this and confirming my suspicion, I genuinely appreciate having someone else looking over things as well :smile:. I have confused `matchLabels` with `labels` that are matched to rather than ones that will be matched to other resources.

I have run the full battery of tests locally as well as completed deployment testing. I believe the production code is stable enough for broader testing. I will be reviewing the test suite and expanding it if necessary.

The only addition I feel I can make to the production code is to validate input for lowercase `RFC-1123` compliance. This will slow the code down because of regex matching. Other than this, I believe we have a feature RC here.
I have added lowercase `RFC-1123` validation for `Volume`, `Claim`, as well as `Storage Class` names. I have tested the validation through the test suite but have not had the opportunity to run deployment testing with them. The full battery of tests is passing locally. This should hopefully catch errors before anything gets handed over to K8s.
Just started some testing in Kubernetes. Deployed the `acking` topology, but when I tried to kill it, I received the following: `Failed to connect to K8s cluster to delete Persistent Volume Claims for topology `acking`. A manual clean-up is required.`

This was a submission without enabling any PVCs for the pods.

**_Edit:_** Disregard, I think I used an old Helm chart that didn't have the new PVC permissions.
No worries, I am not sure if the Helm Chart I edited is minimal enough as far as permissions go - can we get away without the `list` permission? I am performing all of my testing on Minikube in a clean environment (`delete --all`) and perform a manual deployment each time.
So even after adding the permissions, I still get the forbidden failure when deleting. I'll keep investigating.

I did continue testing with the `ondemand`. It was able to create the PVC, but they stay in a pending state. I'm not sure, but might be an issue with how I configured things. One potential issue or design consideration... it seems to only make a single PVC for the entire topology. I was assuming it would be a PVC per pod. Here is the naming convention from the [Spark code](https://github.com/apache/spark/blob/de59e01aa4853ef951da080c0d1908d53d133ebe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala#L76). It seems they have an index `$i` on the end as they loop through?
Could you please try to dump the PVC with the `get persistentvolumeclaims [claim-name] -o yaml` and see if something is off? You can do the same for the Pod's configs.

I am able to `create` and `delete` all the generated PVC's in Minikube, so I think this is a permission-based issue. I to see the status as in `pending` but that is because I either do not have the claims setup for statically backed storage or the dynamically backed storage enabled and configured.
> One potential issue or design consideration... it seems to only make a single PVC for the entire topology. I was assuming it would be a PVC per pod. Here is the naming convention from the [Spark code](https://github.com/apache/spark/blob/de59e01aa4853ef951da080c0d1908d53d133ebe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala#L76). It seems they have an index `$i` on the end as they loop through?

The design is such that a single PVC is generated per dynamically allocated/backed volume for the entire `StatefulSet`. All of the `executor` pods will gain entries added for the `Volume Mounts` and the entire pod template will receive an entry for the shared `Volume`. Since it is a shared `Volume` for the `StatefulSet`, my assumption being this is to support amongst many other things, a shared disk between support containers makes perfect sense. You may allocate as many PVCs as you want for topology, though.

**_Edit:_** If we can use variables in the K8s Pod Template which K8s can resolve, specifically for the Pod number, then this can be done. Say we have the variable `$pod.number` then I would only need to update the following two entries in the configs:
`PersistentVolumeClaim.metadata.name`: ondemand-[topology-name]-[volume-name]-$pod.number
`Volumes.name`: [volume-name]-$pod.number

With this, we make the above field entries with variables in the Pod Templates. When the PVCs are deployed to K8s we add the `$pod.number` suffix to the PVC names on the fly. This is all dependant on the ability to extract the Pod number via a variable.

*Nevertheless, if there is no way we can reference fields in the Pod Template in other fields within the template, I can not see how this can be achieved. There would need to be API access to the individual Pods within the `StatefulSet` to adjust the fields for `Volume Mounts`. Creating the `Volume` and `Persistent Volume Claims` remains a nonissue.*
I can confirm the ability to launch and kill a topology with/without a Pod Template as well as being able to launch a topology with a statically and dynamically backed PVC. The dynamically backed PVC is deleted on kill. I am testing in Minikube on a clean environment (`delete --all`) with the latest PR build.

<details><summary>Without dynamic PVC: kubectl get pod acking-1 -o yaml</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-10T22:36:19Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-7cf596b959
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 7638b318-645b-41e1-b0c9-47e124207a13
  resourceVersion: "852"
  uid: 1e125904-6c32-4c69-a48a-ba85a4136d9b
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--358503390991778356.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=acking8c323447-bce6-4a35-9b2f-7b6e1b9cc46d
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-lbslq
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-lbslq
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - emptyDir: {}
    name: shared-volume
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volume-claim-name
  - name: kube-api-access-lbslq
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-10T22:36:19Z"
    message: '0/1 nodes are available: 1 persistentvolumeclaim "volume-claim-name"
      not found.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>With dynamic PVC: kubectl get pod acking-1 -o yaml</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-10T22:39:57Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-59fdd8c858
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: dc5ae063-672f-4b1c-98cd-0936cd02a1b0
  resourceVersion: "1060"
  uid: cbe65fc2-cddf-496d-b633-68fad946c9b1
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--4583400108540459301.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=acking1f62dd17-33f9-4a72-98e7-daca680e9f16
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-njblm
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-njblm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - emptyDir: {}
    name: shared-volume
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: ondemand-acking-volumenameofchoice
  - name: kube-api-access-njblm
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-10T22:39:57Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>kubectl get persistentvolumeclaims ondemand-acking-volumenameofchoice -o yaml</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2021-11-10T22:39:57Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    onDemand: "true"
    topology: acking
  name: ondemand-acking-volumenameofchoice
  namespace: default
  resourceVersion: "1048"
  uid: 34a9bb66-db51-4abe-bc65-a8b45eef1593
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: storage-class-name
  volumeMode: Block
  volumeName: volumenameofchoice
status:
  phase: Pending
```

</details>

One thought for the `ondemand` approach would be that perhaps we should be leveraging the `VolumeClaimTemplate` feature of a StatefulSet. Here is an [example](https://stackoverflow.com/questions/53927587/kubernetes-multiple-pvc-with-statefulset-for-each-pod-vs-single-pvc-for-all-pods). This would allow for a PVC per pod without having to manually create them. Kubernetes will automatically create with the pod's ordinal number. This might also help answer how we handle the `addContainer` call which can be called when someone rescales a running Heron topology. In this case any new pods would get a new PVC. I think it's ok to leave the PVC if the analytic is scaled down because we will cleanup all PVCs when the topology is killed.

**_Edit:_** This would also allow us to keep most of your logic the same. I do agree there is benefit in having the single shared volume so don't want to lose that. I perhaps was describing things with too strong a focus on the `ondemand` use case.
`VolumeClaimTemplate`s is a cleaner way of handling all this, I shall switch over to using them. I think we can remove the shared PVC volume; there are certain things that should be left to being manually created in Pod Templates. I need to figure out how to rejig the code.

On a side note, we need to fix the CI issues. The `statemgr_unit` test needs to be fixed and we _need_ the ability to manually retrigger the CI pipeline.

**_Edit:_** switching to the `VolumeClaimTemplate` seems to be ~a simple~ an "involved" adjustment with a lot of test updates. I am figuring things out.

We are going to need to drop the `claimName` option and switch to only using only `Volume Claim Templates` going forward. This will allow people to set up a statically backed PV or expect a dynamically backed one using the `storageClassName` option. If nothing is provided for the `storageClassName` a statically backed PV will be expected. This means PVCs will always be generated and deployed whenever anyone uses these CLI options. Thoughts?
@nicknezis: Setup and tear down of all PVCs with/without Pod Templates as well as testing without PVCs successful. I have dumped the `YAML` below to make life easier. Full testing battery passing locally.

**_STATIC:_**

<details><summary>StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-12T01:56:24Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "1421"
  uid: 6e991292-575c-4cbb-b200-3e4a315cf4b6
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--5861445565125076364.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=ackingac5a762c-c2d2-45fc-8b59-f83acfff7564
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        onDemand: "true"
        topology: acking
      name: volumenameofchoice
    spec:
      accessModes:
      - ReadWriteOnce
      - ReadOnlyMany
      resources:
        requests:
          storage: 555Gi
      volumeMode: Block
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-8688f48fb8
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-8688f48fb8
  updatedReplicas: 3
```

</details>

<details><summary>Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-12T01:50:55Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-5f56945486
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 22b719a9-9b7d-4d63-8365-e76b44a50abe
  resourceVersion: "960"
  uid: 35118297-0949-4648-b12f-8b755e583436
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--1817385379878734459.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=ackingff8ff63d-977f-49ba-8521-11375f9c41d6
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-b8b88
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-b8b88
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volumenameofchoice-acking-1
  - emptyDir: {}
    name: shared-volume
  - name: kube-api-access-b8b88
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-12T01:50:55Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>PVC</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
  creationTimestamp: "2021-11-12T01:50:55Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: heron
    onDemand: "true"
    topology: acking
  name: volumenameofchoice-acking-0
  namespace: default
  resourceVersion: "954"
  uid: 0e8c8aec-ce54-410e-8790-5da2238af798
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: standard
  volumeMode: Block
status:
  phase: Pending
```

</details>

**_DYNAMIC:_**

<details><summary>StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-12T01:53:56Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "1212"
  uid: 597877ea-4601-4bf0-80e3-72f9db78ad47
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--6172080214638514375.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=acking6a21ddc3-dbab-4c25-8a54-dcd3b0264b55
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        onDemand: "true"
        topology: acking
      name: volumenameofchoice
    spec:
      accessModes:
      - ReadWriteOnce
      - ReadOnlyMany
      resources:
        requests:
          storage: 555Gi
      storageClassName: storage-class-name
      volumeMode: Block
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-5cbfb4df4c
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-5cbfb4df4c
  updatedReplicas: 3
```

</details>

<details><summary>Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-12T01:53:56Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-5cbfb4df4c
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 597877ea-4601-4bf0-80e3-72f9db78ad47
  resourceVersion: "1205"
  uid: eec86082-2259-4fe7-999a-10f7e58d2e79
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--6172080214638514375.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=acking6a21ddc3-dbab-4c25-8a54-dcd3b0264b55
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-bdwcl
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-bdwcl
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volumenameofchoice-acking-1
  - emptyDir: {}
    name: shared-volume
  - name: kube-api-access-bdwcl
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-12T01:53:56Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>PVC</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2021-11-12T01:53:56Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: heron
    onDemand: "true"
    topology: acking
  name: volumenameofchoice-acking-0
  namespace: default
  resourceVersion: "1190"
  uid: b7a4c403-b749-452b-ac70-16c6a9e158d6
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: storage-class-name
  volumeMode: Block
status:
  phase: Pending
```

</details>
I have updated the usage [documentation](https://github.com/apache/incubator-heron/blob/274f86c3567cd8633eb8c97e968e0c35d06e903c/website2/docs/schedulers-k8s-persistent-volume-claims.md) and merged in the updates. Let us see if TravisCI is in a good mood 🎲 🤞🏼...

**_Edit:_** Updated with potential use cases  [documentation](https://github.com/apache/incubator-heron/blob/c7a97694e111fa3e4bf222fe13ee90d9638c3786/website2/docs/schedulers-k8s-persistent-volume-claims.md).
I figured out why I couldn't delete any topologies. Our Helm chart didn't have the `deletecollection` permission. Added it to the branch.
Apologies for the delay in testing. Now that the other Python issue is resolved, I'm again able to submit and kill topologies. It seems that the scheduler is always attempting to create new PVCs. 

I created the following PVC:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nez-big-storage
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: standard
  volumeMode: Filesystem
```

And then submitted the following: 
```
heron submit heronkube \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.nez-volume-name.readOnly=true \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.nez-volume-name.path=/tmp/nez \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.nez-volume-name.subPath=was/here \
--config-property heron.kubernetes.volumes.persistentVolumeClaim.nez-volume-name.claimname=nez-big-storage \
/heron/examples/heron-api-examples.jar org.apache.heron.examples.api.AckingTopology acking
```

The pre-provisioned PVC would be mounted at the above path `/tmp/nez` with a `volumeMount` named `nez-volume-name`. Not this is the volume name, as opposed to the volume claim name.
The "dynamic" creation of the PVCs would only happen when the special claim name `OnDemand` is used. This would be a special toggle that triggers the logic to dynamically create the PVC with the Claim Template. Only then do we use the other options to populate the Claim Template.

I think we're getting close. Hopefully that explanation helps.

One thing I notice on the Spark config structure that might be worth noting, they use the `mount` and `options` to distinguish the volume mount set of configs and the options that are specific to the type of storage. In our case, so far we are only implementing the PVC approach, but in the future we might also add `hostPath`. I'm not sure if having the `mount` and `options` configuration hierarchy helps make the code cleaner. I can see things still working the way you currently have the naming, so it might not be a big deal.
Thank you for taking the time to look at this!

I think it is important to keep in mind the following key limitations of a `StatefulSet`:

* Pod templates are copied verbatim.
* Volumes are shared with all Pods.
* The same volume mounts are duplicated in matching containers across Pods.
* Only way to have unique volumes per pod is to leverage `volumeClaimTemplates`.

Please see [***Stable Storage***](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/).

If you would like to mount a volume you have already created you need to name it based on the naming convention K8s uses. K8s will initially attempt locating a persistent volume that matches the persistent volume claim, if that fails it attempts to generate a dynamically backed volume.

If I proceed to add volumes to the `StatefulSet`, those volumes will be shared in all Pods. To mount a volume in a Pod you need to have a linked Volume in the Pod's spec. We could have some bash trickery that mounts the unique Volume into a container, but that volume will be shared with all other Pods. The long and the short of it is that we will end up right back where we started with the Volume being shared with all Pods.

It is bad design to utilize the same CLI parameter to mount both unique and shared volumes.

Yes, the case where we mount an existing PVC, this would be a shared PVC accessed by all pods. (i.e. `ReadWriteMany` or `ReadOnlyMany`)

One `ReadOnlyMany` example is you could mount a read only PVC with files such as certificates and certificate authorities. 

For `ReadWriteMany` you can also mount a shared drive for large file access. There are cases in which streaming data is smaller references to larger files. The bolts would have access to the shared volumes to access larger files. The bolts could do file scan actions for specific files based on the metadata only stream.

The case for the dynamic per-pod mounts using the PersistentVolumeClaim Template would be for local checkpointing and log file storage. Currently in each pod there is a `log-files` folder in each container that writes the Spout/Bolt logs to the container's built in storage. With the dynamic PVC mount, one could configure to use a system like Rook/Ceph to provide dynamic storage that does not use the Docker volume which might be more limited.

If there is a better suggestion for the CLI parameters, I'm cool with that. I was merely using the Spark design as a starting point. I see the value in how they structured the subsets of config under `mount` and `options` that supports handling the different storage types (i.e. `nfs`, `pvc`, `hostPath`, etc). But we could totally have a more explicit set of config.

I think it's my fault for discussing both the static mount and dynamic mont. Hopefully I didn't confuse things too much. I really was looking to mirror what Spark provided in terms of the functionality. I think their `OnDemand` and `local` are the only cases which get the per pod specific storage. The other cases, the pods share the same storage.
I am going to go ahead and reintroduce the mounts and volumes for shared PVCs at which point I shall consider this PR completed. As careful as I have been, and as extensive as the test suite is, 1000+ lines of code introduce a lot of moving parts that could go wrong.

> If there is a better suggestion for the CLI parameters, I'm cool with that.

Individual volumes: `heron.kubernetes.volumes.persistentVolumeClaim.volumeNameOfChoice.`
Shared volumes: `heron.kubernetes.volumes.persistentVolumeClaim.shared.volumeNameOfChoice.`

But that is an issue for another time.
I added support for shared `Volume`s via a `claimName` but have not had the chance to complete a full battery of deployment tests.

A Claim name of `OnDemand` will result in PVCs being created. Any other Claim name will only add `Volume`s to the Pod Spec and `Volume Mounts` to the `Executor` Pod and will be replicated between all Pods in the topology.

Documentation is [here](https://github.com/apache/incubator-heron/blob/6a449b178f892f1bd109ba28ead228abc5a52613/website2/docs/schedulers-k8s-persistent-volume-claims.md).

I think we may be able to whittle down the permissions for the PVC to:

```yaml
rules:
- apiGroups: 
 - ""
 resources: 
 - persistentvolumeclaims
 verbs: 
 - create
 - deletecollection
```
I ran some quick tests before I begin separating the `executor`s and the `manager` and everything looks good. `YAML` for each case is below, I can confirm that a shared volume will not add a PVC.

<details><summary>Shared StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-18T17:23:24Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "2717"
  uid: f4e9b2a9-f880-4b80-bd15-7cd54cdd23fb
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0-160636999515090835.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=ackingcf66189f-ca7f-46a3-a0a9-cb41a5d658c4
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
      - name: volumenameofchoice
        persistentVolumeClaim:
          claimName: requested-claim-by-user
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-76494fb95
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-76494fb95
  updatedReplicas: 3
```

</details>

<details><summary>Shared Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-18T17:23:24Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-76494fb95
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: f4e9b2a9-f880-4b80-bd15-7cd54cdd23fb
  resourceVersion: "2718"
  uid: 0e28c20c-e3aa-4f36-a2dd-da063147f5c0
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0-160636999515090835.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=ackingcf66189f-ca7f-46a3-a0a9-cb41a5d658c4
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-8pmpr
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-8pmpr
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - emptyDir: {}
    name: shared-volume
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: requested-claim-by-user
  - name: kube-api-access-8pmpr
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-18T17:23:24Z"
    message: '0/1 nodes are available: 1 persistentvolumeclaim "requested-claim-by-user"
      not found.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>Static StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-18T17:12:52Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "2061"
  uid: 54337851-2bc2-4cf3-bf66-d9e8a5716aa2
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--1619810271492859618.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=acking87b0a985-49ec-4749-aa2d-d7f4aaa736a4
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        onDemand: "true"
        topology: acking
      name: volumenameofchoice
    spec:
      accessModes:
      - ReadWriteOnce
      - ReadOnlyMany
      resources:
        requests:
          storage: 555Gi
      volumeMode: Block
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-6c4c987ddc
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-6c4c987ddc
  updatedReplicas: 3
```

</details>

<details><summary>Static Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-18T17:12:52Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-6c4c987ddc
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 54337851-2bc2-4cf3-bf66-d9e8a5716aa2
  resourceVersion: "2051"
  uid: 027fff42-3f1e-4457-a3f8-6505b256f3af
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--1619810271492859618.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=acking87b0a985-49ec-4749-aa2d-d7f4aaa736a4
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-txlbw
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-txlbw
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volumenameofchoice-acking-1
  - emptyDir: {}
    name: shared-volume
  - name: kube-api-access-txlbw
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-18T17:12:52Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>Static PVC</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
  creationTimestamp: "2021-11-18T17:12:52Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: heron
    onDemand: "true"
    topology: acking
  name: volumenameofchoice-acking-1
  namespace: default
  resourceVersion: "2047"
  uid: 50a226ac-d7cb-44b9-8d55-8ecc9bf8796d
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: standard
  volumeMode: Block
status:
  phase: Pending
```

</details>

<details><summary>Dynamic StatefulSet</summary>

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2021-11-18T17:17:59Z"
  generation: 1
  name: acking
  namespace: default
  resourceVersion: "2412"
  uid: 4263ad61-4d70-4803-9664-5e1cc04c174e
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: heron
      topology: acking
  serviceName: acking
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: heron
        topology: acking
    spec:
      containers:
      - command:
        - sh
        - -c
        - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
          distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--8727299585809965289.tar.gz
          . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
          --topology-name=acking --topology-id=ackinge77c4afd-1609-4616-8344-a3474f0cd2ba
          --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181
          --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml
          --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr
          --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)"
          --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml
          --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824
          --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar
          --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell
          --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/*
          --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
          --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
          --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
          --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
          --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
          --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
          --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
          --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
          --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: var_one
          value: variable one
        - name: var_three
          value: variable three
        - name: var_two
          value: variable two
        image: apache/heron:testbuild
        imagePullPolicy: IfNotPresent
        name: executor
        ports:
        - containerPort: 5555
          name: tcp-port-kept
          protocol: TCP
        - containerPort: 5556
          name: udp-port-kept
          protocol: UDP
        - containerPort: 6001
          name: server
          protocol: TCP
        - containerPort: 6002
          name: tmanager-ctl
          protocol: TCP
        - containerPort: 6003
          name: tmanager-stats
          protocol: TCP
        - containerPort: 6004
          name: shell-port
          protocol: TCP
        - containerPort: 6005
          name: metrics-mgr
          protocol: TCP
        - containerPort: 6006
          name: scheduler
          protocol: TCP
        - containerPort: 6007
          name: metrics-cache-m
          protocol: TCP
        - containerPort: 6008
          name: metrics-cache-s
          protocol: TCP
        - containerPort: 6009
          name: ckptmgr
          protocol: TCP
        resources:
          limits:
            cpu: "3"
            memory: 4Gi
          requests:
            cpu: "3"
            memory: 4Gi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
        - mountPath: path/to/mount
          name: volumenameofchoice
          subPath: sub/path/to/mount
      - image: alpine
        imagePullPolicy: Always
        name: sidecar-container
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /shared_volume
          name: shared-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 0
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 10
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 10
      volumes:
      - emptyDir: {}
        name: shared-volume
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        onDemand: "true"
        topology: acking
      name: volumenameofchoice
    spec:
      accessModes:
      - ReadWriteOnce
      - ReadOnlyMany
      resources:
        requests:
          storage: 555Gi
      storageClassName: storage-class-name
      volumeMode: Block
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: acking-7bfcb94659
  observedGeneration: 1
  replicas: 3
  updateRevision: acking-7bfcb94659
  updatedReplicas: 3
```

</details>

<details><summary>Dynamic Pod</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "8080"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-11-18T17:17:59Z"
  generateName: acking-
  labels:
    app: heron
    controller-revision-hash: acking-7bfcb94659
    statefulset.kubernetes.io/pod-name: acking-1
    topology: acking
  name: acking-1
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: acking
    uid: 4263ad61-4d70-4803-9664-5e1cc04c174e
  resourceVersion: "2406"
  uid: 807570fd-1a38-4a5a-bb5e-cb5fe13acecc
spec:
  containers:
  - command:
    - sh
    - -c
    - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader
      distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0--8727299585809965289.tar.gz
      . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor
      --topology-name=acking --topology-id=ackinge77c4afd-1609-4616-8344-a3474f0cd2ba
      --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron
      --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager
      --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/*
      --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar
      --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml
      --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts=""
      --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME
      --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad
      --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml
      --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/*
      --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance
      --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled
      --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*:
      --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824
      --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/*
      --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003
      --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007
      --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009'
    env:
    - name: HOST
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: var_one
      value: variable one
    - name: var_three
      value: variable three
    - name: var_two
      value: variable two
    image: apache/heron:testbuild
    imagePullPolicy: IfNotPresent
    name: executor
    ports:
    - containerPort: 5555
      name: tcp-port-kept
      protocol: TCP
    - containerPort: 5556
      name: udp-port-kept
      protocol: UDP
    - containerPort: 6001
      name: server
      protocol: TCP
    - containerPort: 6002
      name: tmanager-ctl
      protocol: TCP
    - containerPort: 6003
      name: tmanager-stats
      protocol: TCP
    - containerPort: 6004
      name: shell-port
      protocol: TCP
    - containerPort: 6005
      name: metrics-mgr
      protocol: TCP
    - containerPort: 6006
      name: scheduler
      protocol: TCP
    - containerPort: 6007
      name: metrics-cache-m
      protocol: TCP
    - containerPort: 6008
      name: metrics-cache-s
      protocol: TCP
    - containerPort: 6009
      name: ckptmgr
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 4Gi
      requests:
        cpu: "3"
        memory: 4Gi
    securityContext:
      allowPrivilegeEscalation: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: path/to/mount
      name: volumenameofchoice
      subPath: sub/path/to/mount
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wl7h4
      readOnly: true
  - image: alpine
    imagePullPolicy: Always
    name: sidecar-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /shared_volume
      name: shared-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wl7h4
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: acking-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: acking
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 10
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 10
  volumes:
  - name: volumenameofchoice
    persistentVolumeClaim:
      claimName: volumenameofchoice-acking-1
  - emptyDir: {}
    name: shared-volume
  - name: kube-api-access-wl7h4
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-11-18T17:17:59Z"
    message: '0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable
```

</details>

<details><summary>Dynamic PVC</summary>

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: "2021-11-18T17:17:59Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: heron
    onDemand: "true"
    topology: acking
  name: volumenameofchoice-acking-1
  namespace: default
  resourceVersion: "2395"
  uid: 65a9291e-ea83-41bb-a1ce-56d76574f0f3
spec:
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  resources:
    requests:
      storage: 555Gi
  storageClassName: storage-class-name
  volumeMode: Block
status:
  phase: Pending
```

</details>

Fix patch:
diff --git a/deploy/kubernetes/general/apiserver.yaml b/deploy/kubernetes/general/apiserver.yaml
index 094715a8145..2e6290c26ce 100644
--- a/deploy/kubernetes/general/apiserver.yaml
+++ b/deploy/kubernetes/general/apiserver.yaml
@@ -91,7 +91,7 @@ spec:
               -D heron.uploader.dlog.topologies.namespace.uri=distributedlog://zookeeper:2181/heron
               -D heron.statefulstorage.classname=org.apache.heron.statefulstorage.dlog.DlogStorage
               -D heron.statefulstorage.dlog.namespace.uri=distributedlog://zookeeper:2181/heron
-              -D heron.kubernetes.pod.template.configmap.disabled=false
+              -D heron.kubernetes.pod.template.disabled=false
               -D heron.kubernetes.persistent.volume.claims.cli.disabled=false
 
 ---
diff --git a/deploy/kubernetes/helm/templates/tools.yaml b/deploy/kubernetes/helm/templates/tools.yaml
index c776e49cfd0..f418779d676 100644
--- a/deploy/kubernetes/helm/templates/tools.yaml
+++ b/deploy/kubernetes/helm/templates/tools.yaml
@@ -158,7 +158,7 @@ spec:
               -D heron.class.repacking.algorithm=org.apache.heron.packing.binpacking.FirstFitDecreasingPacking
               {{- end }}
               -D heron.kubernetes.resource.request.mode={{ .Values.topologyResourceRequestMode }}
-              -D heron.kubernetes.pod.template.configmap.disabled={{ .Values.disablePodTemplates }}
+              -D heron.kubernetes.pod.template.disabled={{ .Values.disablePodTemplates }}
               -D heron.kubernetes.persistent.volume.claims.cli.disabled={{ .Values.disablePersistentVolumeMountsCLI }}
           envFrom:
             - configMapRef:
@@ -297,6 +297,7 @@ rules:
   - patch
   - update
   - watch
+  - deletecollection
 - apiGroups:
   - ""
   resources:
diff --git a/deploy/kubernetes/minikube/apiserver.yaml b/deploy/kubernetes/minikube/apiserver.yaml
index 53d879ab515..ce20c90f111 100644
--- a/deploy/kubernetes/minikube/apiserver.yaml
+++ b/deploy/kubernetes/minikube/apiserver.yaml
@@ -82,7 +82,7 @@ spec:
               -D heron.uploader.dlog.topologies.namespace.uri=distributedlog://zookeeper:2181/heronbkdl
               -D heron.statefulstorage.classname=org.apache.heron.statefulstorage.dlog.DlogStorage
               -D heron.statefulstorage.dlog.namespace.uri=distributedlog://zookeeper:2181/heronbkdl
-              -D heron.kubernetes.pod.template.configmap.disabled=false
+              -D heron.kubernetes.pod.template.disabled=false
               -D heron.kubernetes.persistent.volume.claims.cli.disabled=false
 
 ---
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesConstants.java b/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesConstants.java
index 023d8bccde9..c89a0c694f9 100644
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesConstants.java
+++ b/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesConstants.java
@@ -36,6 +36,7 @@ private KubernetesConstants() {
   public static final String CPU = "cpu";
 
   public static final String EXECUTOR_NAME = "executor";
+  public static final String MANAGER_NAME = "manager";
 
   // container env constants
   public static final String ENV_HOST = "HOST";
@@ -89,6 +90,9 @@ private KubernetesConstants() {
   public static final String JOB_LINK =
       "/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#/pod";
 
+  protected static final String JSON_PATCH_STATEFUL_SET_REPLICAS_FORMAT =
+      "[{\"op\":\"replace\",\"path\":\"/spec/replicas\",\"value\":%d}]";
+
   public static final Pattern VALID_POD_NAME_REGEX =
       Pattern.compile("[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*",
           Pattern.CASE_INSENSITIVE);
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesContext.java b/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesContext.java
index 40b76187cdc..83aa0b20d16 100644
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesContext.java
+++ b/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/KubernetesContext.java
@@ -86,11 +86,11 @@ public enum KubernetesResourceRequestMode {
   public static final String KUBERNETES_VOLUME_AWS_EBS_FS_TYPE =
       "heron.kubernetes.volume.awsElasticBlockStore.fsType";
 
-  // pod template configmap
-  public static final String KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME =
-      "heron.kubernetes.pod.template.configmap.name";
-  public static final String KUBERNETES_POD_TEMPLATE_CONFIGMAP_DISABLED =
-      "heron.kubernetes.pod.template.configmap.disabled";
+  // Pod Template ConfigMap: heron.kubernetes.[executor | manager].pod.template
+  public static final String KUBERNETES_POD_TEMPLATE_LOCATION =
+      "heron.kubernetes.%s.pod.template";
+  public static final String KUBERNETES_POD_TEMPLATE_DISABLED =
+      "heron.kubernetes.pod.template.disabled";
 
   // container mount volume mount keys
   public static final String KUBERNETES_CONTAINER_VOLUME_MOUNT_NAME =
@@ -116,9 +116,15 @@ public enum KubernetesResourceRequestMode {
   // Persistent Volume Claims
   public static final String KUBERNETES_PERSISTENT_VOLUME_CLAIMS_CLI_DISABLED =
       "heron.kubernetes.persistent.volume.claims.cli.disabled";
-  // heron.kubernetes.volumes.persistentVolumeClaim.VOLUME_NAME.OPTION=OPTION_VALUE
+  // heron.kubernetes.[executor | manager].volumes.persistentVolumeClaim.VOLUME_NAME.OPTION=VALUE
   public static final String KUBERNETES_VOLUME_CLAIM_PREFIX =
-      "heron.kubernetes.volumes.persistentVolumeClaim.";
+      "heron.kubernetes.%s.volumes.persistentVolumeClaim.";
+  // heron.kubernetes.[executor | manager].limits.OPTION=VALUE
+  public static final String KUBERNETES_RESOURCE_LIMITS_PREFIX =
+      "heron.kubernetes.%s.limits.";
+  // heron.kubernetes.[executor | manager].requests.OPTION=VALUE
+  public static final String KUBERNETES_RESOURCE_REQUESTS_PREFIX =
+      "heron.kubernetes.%s.requests.";
 
   private KubernetesContext() {
   }
@@ -189,12 +195,14 @@ static String getContainerVolumeMountPath(Config config) {
     return config.getStringValue(KUBERNETES_CONTAINER_VOLUME_MOUNT_PATH);
   }
 
-  public static String getPodTemplateConfigMapName(Config config) {
-    return config.getStringValue(KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME);
+  public static String getPodTemplateConfigMapName(Config config, boolean isExecutor) {
+    final String key = String.format(KUBERNETES_POD_TEMPLATE_LOCATION,
+        isExecutor ? KubernetesConstants.EXECUTOR_NAME : KubernetesConstants.MANAGER_NAME);
+    return config.getStringValue(key);
   }
 
-  public static boolean getPodTemplateConfigMapDisabled(Config config) {
-    final String disabled = config.getStringValue(KUBERNETES_POD_TEMPLATE_CONFIGMAP_DISABLED);
+  public static boolean getPodTemplateDisabled(Config config) {
+    final String disabled = config.getStringValue(KUBERNETES_POD_TEMPLATE_DISABLED);
     return "true".equalsIgnoreCase(disabled);
   }
 
@@ -222,6 +230,18 @@ public static Map<String, String> getPodSecretKeyRefs(Config config) {
     return getConfigItemsByPrefix(config, KUBERNETES_POD_SECRET_KEY_REF_PREFIX);
   }
 
+  public static Map<String, String> getResourceLimits(Config config, boolean isExecutor) {
+    final String key = String.format(KUBERNETES_RESOURCE_LIMITS_PREFIX,
+        isExecutor ? KubernetesConstants.EXECUTOR_NAME : KubernetesConstants.MANAGER_NAME);
+    return getConfigItemsByPrefix(config, key);
+  }
+
+  public static Map<String, String> getResourceRequests(Config config, boolean isExecutor) {
+    final String key = String.format(KUBERNETES_RESOURCE_REQUESTS_PREFIX,
+        isExecutor ? KubernetesConstants.EXECUTOR_NAME : KubernetesConstants.MANAGER_NAME);
+    return getConfigItemsByPrefix(config, key);
+  }
+
   public static boolean getPersistentVolumeClaimDisabled(Config config) {
     final String disabled = config.getStringValue(KUBERNETES_PERSISTENT_VOLUME_CLAIMS_CLI_DISABLED);
     return "true".equalsIgnoreCase(disabled);
@@ -231,15 +251,18 @@ public static boolean getPersistentVolumeClaimDisabled(Config config) {
    * Collects parameters form the <code>CLI</code> and generates a mapping between <code>Volumes</code>
    * and their configuration <code>key-value</code> pairs.
    * @param config Contains the configuration options collected from the <code>CLI</code>.
+   * @param isExecutor Flag used to collect CLI commands for the <code>executor</code> and <code>manager</code>.
    * @return A mapping between <code>Volumes</code> and their configuration <code>key-value</code> pairs.
    * Will return an empty list if there are no Volume Claim Templates to be generated.
    */
   public static Map<String, Map<KubernetesConstants.VolumeClaimTemplateConfigKeys, String>>
-      getVolumeClaimTemplates(Config config) {
+      getVolumeClaimTemplates(Config config, boolean isExecutor) {
     final Logger LOG = Logger.getLogger(V1Controller.class.getName());
 
-    final Set<String> completeConfigParam = getConfigKeys(config, KUBERNETES_VOLUME_CLAIM_PREFIX);
-    final int prefixLength = KUBERNETES_VOLUME_CLAIM_PREFIX.length();
+    final String prefixKey = String.format(KUBERNETES_VOLUME_CLAIM_PREFIX,
+        isExecutor ? KubernetesConstants.EXECUTOR_NAME : KubernetesConstants.MANAGER_NAME);
+    final Set<String> completeConfigParam = getConfigKeys(config, prefixKey);
+    final int prefixLength = prefixKey.length();
     final int volumeNameIdx = 0;
     final int optionIdx = 1;
     final Matcher matcher = KubernetesConstants.VALID_LOWERCASE_RFC_1123_REGEX.matcher("");
diff --git a/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/V1Controller.java b/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/V1Controller.java
index e6fc819e31c..96793b6d969 100644
--- a/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/V1Controller.java
+++ b/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/V1Controller.java
@@ -99,13 +99,10 @@ public class V1Controller extends KubernetesController {
   private final AppsV1Api appsClient;
   private final CoreV1Api coreClient;
 
-  private Map<String, Map<KubernetesConstants.VolumeClaimTemplateConfigKeys, String>>
-      persistentVolumeClaimConfigs = null;
-
   V1Controller(Config configuration, Config runtimeConfiguration) {
     super(configuration, runtimeConfiguration);
 
-    isPodTemplateDisabled = KubernetesContext.getPodTemplateConfigMapDisabled(configuration);
+    isPodTemplateDisabled = KubernetesContext.getPodTemplateDisabled(configuration);
     LOG.log(Level.WARNING, String.format("Custom Pod Templates are %s",
         isPodTemplateDisabled ? "DISABLED" : "ENABLED"));
 
@@ -124,7 +121,7 @@ public class V1Controller extends KubernetesController {
   boolean submit(PackingPlan packingPlan) {
     final String topologyName = getTopologyName();
     if (!topologyName.equals(topologyName.toLowerCase())) {
-      throw new TopologySubmissionException("K8S scheduler does not allow upper case topologies.");
+      throw new TopologySubmissionException("K8S scheduler does not allow upper case topology's.");
     }
 
     final Resource containerResource = getContainerResource(packingPlan);
@@ -138,28 +135,19 @@ boolean submit(PackingPlan packingPlan) {
       throw new TopologySubmissionException(e.getMessage());
     }
 
-    // Get and then create Persistent Volume Claims from the CLI.
-    persistentVolumeClaimConfigs =
-        KubernetesContext.getVolumeClaimTemplates(getConfiguration());
-    if (KubernetesContext.getPersistentVolumeClaimDisabled(getConfiguration())
-        && !persistentVolumeClaimConfigs.isEmpty()) {
-      final String message =
-          String.format("Configuring Persistent Volume Claim from CLI is disabled: '%s'",
-              topologyName);
-      LOG.log(Level.WARNING, message);
-      throw new TopologySubmissionException(message);
-    }
-
-    // find the max number of instances in a container so we can open
+    // Find the max number of instances in a container so that we can open
     // enough ports if remote debugging is enabled.
     int numberOfInstances = 0;
     for (PackingPlan.ContainerPlan containerPlan : packingPlan.getContainers()) {
       numberOfInstances = Math.max(numberOfInstances, containerPlan.getInstances().size());
     }
-    final V1StatefulSet statefulSet = createStatefulSet(containerResource, numberOfInstances);
+    final V1StatefulSet executors = createStatefulSet(containerResource, numberOfInstances, true);
+    final V1StatefulSet manager = createStatefulSet(containerResource, numberOfInstances, false);
 
     try {
-      appsClient.createNamespacedStatefulSet(getNamespace(), statefulSet, null,
+      appsClient.createNamespacedStatefulSet(getNamespace(), executors, null,
+              null, null);
+      appsClient.createNamespacedStatefulSet(getNamespace(), manager, null,
               null, null);
     } catch (ApiException e) {
       final String message = String.format("Error creating topology: %s%n", e.getResponseBody());
@@ -173,7 +161,7 @@ boolean submit(PackingPlan packingPlan) {
   @Override
   boolean killTopology() {
     removePersistentVolumeClaims();
-    deleteStatefulSet();
+    deleteStatefulSets();
     deleteService();
     return true;
   }
@@ -186,9 +174,8 @@ boolean killTopology() {
   @Override
   boolean restart(int shardId) {
     try {
-      coreClient.deleteCollectionNamespacedPod(getNamespace(), null, null, null, null,
-          0, createTopologySelectorLabels(), null, null, null, null,
-          null, null, null);
+      coreClient.deleteCollectionNamespacedPod(getNamespace(), null, null, null, null, 0,
+          createTopologySelectorLabels(), null, null, null, null, null, null, null);
       LOG.log(Level.WARNING, String.format("Restarting topology '%s'...", getTopologyName()));
     } catch (ApiException e) {
       LOG.log(Level.SEVERE, String.format("Failed to restart topology '%s'...", getTopologyName()));
@@ -245,14 +232,14 @@ public void removeContainers(Set<PackingPlan.ContainerPlan> containersToRemove)
 
   private void patchStatefulSetReplicas(int replicas) throws ApiException {
     final String body =
-            String.format(JSON_PATCH_STATEFUL_SET_REPLICAS_FORMAT,
+            String.format(KubernetesConstants.JSON_PATCH_STATEFUL_SET_REPLICAS_FORMAT,
                     replicas);
     final V1Patch patch = new V1Patch(body);
 
     PatchUtils.patch(V1StatefulSet.class,
         () ->
           appsClient.patchNamespacedStatefulSetCall(
-            getTopologyName(),
+            getStatefulSetName(true),
             getNamespace(),
             patch,
             null,
@@ -264,11 +251,8 @@ private void patchStatefulSetReplicas(int replicas) throws ApiException {
         appsClient.getApiClient());
   }
 
-  private static final String JSON_PATCH_STATEFUL_SET_REPLICAS_FORMAT =
-          "[{\"op\":\"replace\",\"path\":\"/spec/replicas\",\"value\":%d}]";
-
   V1StatefulSet getStatefulSet() throws ApiException {
-    return appsClient.readNamespacedStatefulSet(getTopologyName(), getNamespace(),
+    return appsClient.readNamespacedStatefulSet(getStatefulSetName(true), getNamespace(),
         null, null, null);
   }
 
@@ -307,18 +291,19 @@ void deleteService() {
             + "] in namespace [" + getNamespace() + "] is deleted.");
   }
 
-  void deleteStatefulSet() {
-    try (Response response = appsClient.deleteNamespacedStatefulSetCall(getTopologyName(),
-          getNamespace(), null, null, 0, null,
-          KubernetesConstants.DELETE_OPTIONS_PROPAGATION_POLICY, null, null).execute()) {
+  void deleteStatefulSets() {
+    try (Response response = appsClient.deleteCollectionNamespacedStatefulSetCall(getNamespace(),
+        null, null, null, null, null, createTopologySelectorLabels(), null, null, null, null, null,
+            null, null, null)
+        .execute()) {
 
       if (!response.isSuccessful()) {
         if (response.code() == HTTP_NOT_FOUND) {
-          LOG.log(Level.WARNING, "Tried to delete a non-existent StatefulSet for Topology: "
+          LOG.log(Level.WARNING, "Tried to delete a non-existent StatefulSets for Topology: "
                   + getTopologyName());
           return;
         }
-        LOG.log(Level.SEVERE, "Error when deleting the StatefulSet of the job ["
+        LOG.log(Level.SEVERE, "Error when deleting the StatefulSets of the job ["
                 + getTopologyName() + "] in namespace [" + getNamespace() + "]");
         LOG.log(Level.SEVERE, "Error killing topology message: " + response.message());
         KubernetesUtils.logResponseBodyIfPresent(LOG, response);
@@ -333,16 +318,24 @@ void deleteStatefulSet() {
         return;
       }
       throw new TopologyRuntimeManagementException("Error deleting topology ["
-              + getTopologyName() + "] Kubernetes StatefulSet", e);
+              + getTopologyName() + "] Kubernetes StatefulSets", e);
     } catch (IOException e) {
       throw new TopologyRuntimeManagementException("Error deleting topology ["
-              + getTopologyName() + "] Kubernetes StatefulSet", e);
+              + getTopologyName() + "] Kubernetes StatefulSets", e);
     }
     LOG.log(Level.INFO, "StatefulSet for the Job [" + getTopologyName()
             + "] in namespace [" + getNamespace() + "] is deleted.");
   }
 
-  protected List<String> getExecutorCommand(String containerId, int numOfInstances) {
+  /**
+   * Generates the command to start Heron within the <code>container</code>.
+   * @param containerId Passed down to <>SchedulerUtils</> to generate executor command.
+   * @param numOfInstances Used to configure the debugging ports.
+   * @param isExecutor Flag used to generate the correct <code>shard_id</code>.
+   * @return The complete command to start Heron in a <code>container</code>.
+   */
+  protected List<String> getExecutorCommand(String containerId, int numOfInstances,
+                                            boolean isExecutor) {
     final Config configuration = getConfiguration();
     final Config runtimeConfiguration = getRuntimeConfiguration();
     final Map<ExecutorPort, String> ports =
@@ -370,13 +363,22 @@ protected List<String> getExecutorCommand(String containerId, int numOfInstances
         "-c",
         KubernetesUtils.getConfCommand(configuration)
             + " && " + KubernetesUtils.getFetchCommand(configuration, runtimeConfiguration)
-            + " && " + setShardIdEnvironmentVariableCommand()
+            + " && " + setShardIdEnvironmentVariableCommand(isExecutor)
             + " && " + String.join(" ", executorCommand)
     );
   }
 
-  private static String setShardIdEnvironmentVariableCommand() {
-    return String.format("%s=${POD_NAME##*-} && echo shardId=${%s}", ENV_SHARD_ID, ENV_SHARD_ID);
+  /**
+   * Configures the <code>shard_id</code> for the Heron container based on whether it is an <code>executor</code>
+   * or <code>manager</code>. <code>executor</code> IDs are [1 - n) and the <code>manager</code> IDs start at 0.
+   * @param isExecutor Switch flag to generate correct command.
+   * @return The command required to put the Heron instance in <code>executor</code> or <code>manager</code> mode.
+   */
+  @VisibleForTesting
+  protected static String setShardIdEnvironmentVariableCommand(boolean isExecutor) {
+    final String pattern = String.format("%%s=%s && echo shardId=${%%s}",
+        isExecutor ? "$((${POD_NAME##*-} + 1))" : "${POD_NAME##*-}");
+    return String.format(pattern, ENV_SHARD_ID, ENV_SHARD_ID);
   }
 
   private V1Service createTopologyService() {
@@ -401,21 +403,45 @@ private V1Service createTopologyService() {
     return service;
   }
 
-  private V1StatefulSet createStatefulSet(Resource containerResource, int numberOfInstances) {
+  /**
+   * Creates and configures the <code>StatefulSet</code> which the topology's <code>executor</code>s will run in.
+   * @param containerResource Passed down to configure the <code>executor</code> resource limits.
+   * @param numberOfInstances Used to configure the execution command and ports for the <code>executor</code>.
+   * @param isExecutor Flag used to configure components specific to <code>executor</code> and <code>manager</code>.
+   * @return A fully configured <code>StatefulSet</code> for the topology's <code>executors</code>.
+   */
+  private V1StatefulSet createStatefulSet(Resource containerResource, int numberOfInstances,
+                                          boolean isExecutor) {
     final String topologyName = getTopologyName();
     final Config runtimeConfiguration = getRuntimeConfiguration();
 
+    // Get and then create Persistent Volume Claims from the CLI.
+    final Map<String, Map<KubernetesConstants.VolumeClaimTemplateConfigKeys, String>> configsPVC =
+        KubernetesContext.getVolumeClaimTemplates(getConfiguration(), isExecutor);
+    if (KubernetesContext.getPersistentVolumeClaimDisabled(getConfiguration())
+        && !configsPVC.isEmpty()) {
+      final String message =
+          String.format("'%s': Configuring Persistent Volume Claim from CLI is disabled",
+              topologyName);
+      LOG.log(Level.WARNING, message);
+      throw new TopologySubmissionException(message);
+    }
+
     final V1StatefulSet statefulSet = new V1StatefulSet();
 
     // Setup StatefulSet's metadata.
     final V1ObjectMeta objectMeta = new V1ObjectMeta()
-        .name(topologyName);
+        .name(getStatefulSetName(isExecutor))
+        .labels(getPodLabels(topologyName));
     statefulSet.setMetadata(objectMeta);
 
-    // Create the stateful set spec.
+    // Create the StatefulSet Spec.
+    // Reduce replica count by one for Executors and set to one for Manager.
+    final int replicasCount =
+        isExecutor ? Runtime.numContainers(runtimeConfiguration).intValue() - 1 : 1;
     final V1StatefulSetSpec statefulSetSpec = new V1StatefulSetSpec()
         .serviceName(topologyName)
-        .replicas(Runtime.numContainers(runtimeConfiguration).intValue());
+        .replicas(replicasCount);
 
     // Parallel pod management tells the StatefulSet controller to launch or terminate
     // all Pods in parallel, and not to wait for Pods to become Running and Ready or completely
@@ -428,10 +454,10 @@ private V1StatefulSet createStatefulSet(Resource containerResource, int numberOf
         .matchLabels(getPodMatchLabels(topologyName));
     statefulSetSpec.setSelector(selector);
 
-    // create a pod template
-    final V1PodTemplateSpec podTemplateSpec = loadPodFromTemplate();
+    // Create a Pod Template.
+    final V1PodTemplateSpec podTemplateSpec = loadPodFromTemplate(isExecutor);
 
-    // set up pod meta
+    // Set up Pod Metadata.
     final V1ObjectMeta templateMetaData = new V1ObjectMeta().labels(getPodLabels(topologyName));
     Map<String, String> annotations = new HashMap<>();
     annotations.putAll(getPodAnnotations());
@@ -439,27 +465,23 @@ private V1StatefulSet createStatefulSet(Resource containerResource, int numberOf
     templateMetaData.setAnnotations(annotations);
     podTemplateSpec.setMetadata(templateMetaData);
 
-    final List<String> command = getExecutorCommand("$" + ENV_SHARD_ID, numberOfInstances);
-    configurePodSpec(podTemplateSpec, command, containerResource, numberOfInstances);
+    configurePodSpec(podTemplateSpec, containerResource, numberOfInstances, isExecutor, configsPVC);
 
     statefulSetSpec.setTemplate(podTemplateSpec);
 
     statefulSet.setSpec(statefulSetSpec);
 
-    statefulSetSpec.setVolumeClaimTemplates(
-        createPersistentVolumeClaims(persistentVolumeClaimConfigs));
+    statefulSetSpec.setVolumeClaimTemplates(createPersistentVolumeClaims(configsPVC));
 
     return statefulSet;
   }
 
   private Map<String, String> getPodAnnotations() {
-    Config config = getConfiguration();
-    return KubernetesContext.getPodAnnotations(config);
+    return KubernetesContext.getPodAnnotations(getConfiguration());
   }
 
   private Map<String, String> getServiceAnnotations() {
-    Config config = getConfiguration();
-    return KubernetesContext.getServiceAnnotations(config);
+    return KubernetesContext.getServiceAnnotations(getConfiguration());
   }
 
   private Map<String, String> getPrometheusAnnotations() {
@@ -490,51 +512,60 @@ private Map<String, String> getServiceLabels() {
     return KubernetesContext.getServiceLabels(getConfiguration());
   }
 
-  private void configurePodSpec(final V1PodTemplateSpec podTemplateSpec,
-                                List<String> executorCommand,
-                                Resource resource,
-                                int numberOfInstances) {
+  /**
+   * Configures the <code>Pod Spec</code> section of the <code>StatefulSet</code>. The <code>Heron</code> container
+   * will be configured to allow it to function but other supplied containers are loaded verbatim.
+   * @param podTemplateSpec The <code>Pod Template Spec</code> section to update.
+   * @param resource Passed down to configure the resource limits.
+   * @param numberOfInstances Passed down to configure the ports.
+   * @param isExecutor Flag used to configure components specific to <code>Executor</code> and <code>Manager</code>.
+   * @param configPVC <code>Persistent Volume Claim</code> configurations options.
+   */
+  private void configurePodSpec(final V1PodTemplateSpec podTemplateSpec, Resource resource,
+      int numberOfInstances, boolean isExecutor,
+      final Map<String, Map<KubernetesConstants.VolumeClaimTemplateConfigKeys, String>> configPVC) {
     if (podTemplateSpec.getSpec() == null) {
       podTemplateSpec.setSpec(new V1PodSpec());
     }
     final V1PodSpec podSpec = podTemplateSpec.getSpec();
 
-    // set the termination period to 0 so pods can be deleted quickly
+    // Set the termination period to 0 so pods can be deleted quickly
     podSpec.setTerminationGracePeriodSeconds(0L);
 
-    // set the pod tolerations so pods are rescheduled when nodes go down
+    // Set the pod tolerations so pods are rescheduled when nodes go down
     // https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions
     configureTolerations(podSpec);
 
-    // Get <executor> container and discard all others.
-    final String executorName = KubernetesConstants.EXECUTOR_NAME;
-    V1Container executorContainer = null;
+    // Get <Heron> container and ignore all others.
+    final String containerName =
+        isExecutor ? KubernetesConstants.EXECUTOR_NAME : KubernetesConstants.MANAGER_NAME;
+    V1Container heronContainer = null;
     List<V1Container> containers = podSpec.getContainers();
     if (containers != null) {
       for (V1Container container : containers) {
         final String name = container.getName();
-        if (name != null && name.equals(executorName)) {
-          if (executorContainer != null) {
+        if (name != null && name.equals(containerName)) {
+          if (heronContainer != null) {
             throw new TopologySubmissionException(
-                String.format("Multiple configurations found for %s container", executorName));
+                String.format("Multiple configurations found for '%s' container", containerName));
           }
-          executorContainer = container;
+          heronContainer = container;
         }
       }
     } else {
       containers = new LinkedList<>();
     }
 
-    if (executorContainer == null) {
-      executorContainer = new V1Container().name(executorName);
-      containers.add(executorContainer);
+    if (heronContainer == null) {
+      heronContainer = new V1Container().name(containerName);
+      containers.add(heronContainer);
     }
 
-    if (!persistentVolumeClaimConfigs.isEmpty()) {
-      configurePodWithPersistentVolumeClaimVolumesAndMounts(podSpec, executorContainer);
+    if (!configPVC.isEmpty()) {
+      configurePodWithPersistentVolumeClaimVolumesAndMounts(podSpec, heronContainer, configPVC);
     }
 
-    configureExecutorContainer(executorCommand, resource, numberOfInstances, executorContainer);
+    configureHeronContainer(resource, numberOfInstances, heronContainer, isExecutor);
 
     podSpec.setContainers(containers);
 
@@ -543,6 +574,10 @@ private void configurePodSpec(final V1PodTemplateSpec podTemplateSpec,
     mountSecretsAsVolumes(podSpec);
   }
 
+  /**
+   * Adds <code>tolerations</code> to the <code>Pod Spec</code> with Heron's values taking precedence.
+   * @param spec <code>Pod Spec</code> to be configured.
+   */
   @VisibleForTesting
   protected void configureTolerations(final V1PodSpec spec) {
     KubernetesUtils.V1ControllerUtils<V1Toleration> utils =
@@ -553,6 +588,10 @@ protected void configureTolerations(final V1PodSpec spec) {
     );
   }
 
+  /**
+   * Generates a list of <code>tolerations</code> which Heron requires.
+   * @return A list of configured <code>tolerations</code>.
+   */
   @VisibleForTesting
   protected static List<V1Toleration> getTolerations() {
     final List<V1Toleration> tolerations = new ArrayList<>();
@@ -569,6 +608,10 @@ protected static List<V1Toleration> getTolerations() {
     return tolerations;
   }
 
+  /**
+   * Adds volume to the <code>Pod Spec</code> that Heron requires. Heron's values taking precedence.
+   * @param spec <code>Pod Spec</code> to be configured.
+   */
   @VisibleForTesting
   protected void addVolumesIfPresent(final V1PodSpec spec) {
     final Config config = getConfiguration();
@@ -606,15 +649,24 @@ private void mountSecretsAsVolumes(V1PodSpec podSpec) {
     }
   }
 
-  private void configureExecutorContainer(List<String> executorCommand, Resource resource,
-                                          int numberOfInstances, final V1Container container) {
+  /**
+   * Configures the <code>Heron</code> container with values for parameters Heron requires for functioning.
+   * @param resource Resource limits.
+   * @param numberOfInstances Required number of <code>executor</code> containers which is used to configure ports.
+   * @param container The <code>executor</code> container to be configured.
+   * @param isExecutor Flag indicating whether to set a <code>executor</code> or <code>manager</code> command.
+   */
+  private void configureHeronContainer(Resource resource, int numberOfInstances,
+                                       final V1Container container, boolean isExecutor) {
     final Config configuration = getConfiguration();
 
     // Set up the container images.
     container.setImage(KubernetesContext.getExecutorDockerImage(configuration));
 
     // Set up the container command.
-    container.setCommand(executorCommand);
+    final List<String> command =
+        getExecutorCommand("$" + ENV_SHARD_ID, numberOfInstances, isExecutor);
+    container.setCommand(command);
 
     if (KubernetesContext.hasImagePullPolicy(configuration)) {
       container.setImagePullPolicy(KubernetesContext.getKubernetesImagePullPolicy(configuration));
@@ -627,7 +679,7 @@ private void configureExecutorContainer(List<String> executorCommand, Resource r
     setSecretKeyRefs(container);
 
     // Set container resources
-    configureContainerResources(container, configuration, resource);
+    configureContainerResources(container, configuration, resource, isExecutor);
 
     // Set container ports.
     final boolean debuggingEnabled =
@@ -639,30 +691,60 @@ private void configureExecutorContainer(List<String> executorCommand, Resource r
     mountVolumeIfPresent(container);
   }
 
+  /**
+   * Configures the resources in the <code>container</code> with values in the <code>config</code> taking precedence.
+   * @param container The <code>container</code> to be configured.
+   * @param configuration The <code>Config</code> object to check if a resource request needs to be set.
+   * @param resource User defined resources limits from input.
+   * @param isExecutor
+   */
   @VisibleForTesting
   protected void configureContainerResources(final V1Container container,
-                                             final Config configuration, final Resource resource) {
+                                             final Config configuration, final Resource resource,
+                                             boolean isExecutor) {
     if (container.getResources() == null) {
       container.setResources(new V1ResourceRequirements());
     }
     final V1ResourceRequirements resourceRequirements = container.getResources();
 
-    // Configure resource limits. Deduplicate on limit name with user values taking precedence.
+    // Collect Limits and Requests from CLI.
+    final Map<String, Quantity> limitsCLI = createResourcesRequirement(
+        KubernetesContext.getResourceLimits(configuration, isExecutor));
+    final Map<String, Quantity> requestsCLI = createResourcesRequirement(
+        KubernetesContext.getResourceRequests(configuration, isExecutor));
+
     if (resourceRequirements.getLimits() == null) {
       resourceRequirements.setLimits(new HashMap<>());
     }
+
+    // Set Limits and Resources from CLI <if> available, <else> use Configs. Deduplicate on name
+    // with precedence [1] CLI, [2] Config.
     final Map<String, Quantity> limits = resourceRequirements.getLimits();
-    limits.put(KubernetesConstants.MEMORY,
-        Quantity.fromString(KubernetesUtils.Megabytes(
-            resource.getRam())));
-    limits.put(KubernetesConstants.CPU,
-        Quantity.fromString(Double.toString(roundDecimal(
-            resource.getCpu(), 3))));
+    final Quantity limitCPU = limitsCLI.getOrDefault(KubernetesConstants.CPU,
+        Quantity.fromString(Double.toString(roundDecimal(resource.getCpu(), 3))));
+    final Quantity limitMEMORY = limitsCLI.getOrDefault(KubernetesConstants.MEMORY,
+        Quantity.fromString(KubernetesUtils.Megabytes(resource.getRam())));
+
+    limits.put(KubernetesConstants.MEMORY, limitMEMORY);
+    limits.put(KubernetesConstants.CPU, limitCPU);
 
     // Set the Kubernetes container resource request.
+    // Order: [1] CLI, [2] EQUAL_TO_LIMIT, [3] NOT_SET
     KubernetesContext.KubernetesResourceRequestMode requestMode =
         KubernetesContext.getKubernetesRequestMode(configuration);
-    if (requestMode == KubernetesContext.KubernetesResourceRequestMode.EQUAL_TO_LIMIT) {
+    if (!requestsCLI.isEmpty()) {
+      if (resourceRequirements.getRequests() == null) {
+        resourceRequirements.setRequests(new HashMap<>());
+      }
+      final Map<String, Quantity> requests = resourceRequirements.getRequests();
+
+      if (requestsCLI.containsKey(KubernetesConstants.MEMORY)) {
+        requests.put(KubernetesConstants.MEMORY, requestsCLI.get(KubernetesConstants.MEMORY));
+      }
+      if (requestsCLI.containsKey(KubernetesConstants.CPU)) {
+        requests.put(KubernetesConstants.CPU, requestsCLI.get(KubernetesConstants.CPU));
+      }
+    } else if (requestMode == KubernetesContext.KubernetesResourceRequestMode.EQUAL_TO_LIMIT) {
       LOG.log(Level.CONFIG, "Setting K8s Request equal to Limit");
       resourceRequirements.setRequests(limits);
     } else {
@@ -671,6 +753,38 @@ protected void configureContainerResources(final V1Container container,
     container.setResources(resourceRequirements);
   }
 
+  /**
+   * Creates <code>Resource Requirements</code> from a Map of <code>Config</code> items for <code>CPU</code>
+   * and <code>Memory</code>.
+   * @param configs <code>Configs</code> to be parsed for configuration.
+   * @return Configured <code>Resource Requirements</code>. An <code>empty</code> map will be returned
+   * if there are no <code>configs</code>.
+   */
+  @VisibleForTesting
+  protected Map<String, Quantity> createResourcesRequirement(Map<String, String> configs) {
+    final Map<String, Quantity> requirements = new HashMap<>();
+
+    if (configs == null || configs.isEmpty()) {
+      return requirements;
+    }
+
+    final String memoryLimit = configs.get(KubernetesConstants.MEMORY);
+    if (memoryLimit != null && !memoryLimit.isEmpty()) {
+      requirements.put(KubernetesConstants.MEMORY, Quantity.fromString(memoryLimit));
+    }
+    final String cpuLimit = configs.get(KubernetesConstants.CPU);
+    if (cpuLimit != null && !cpuLimit.isEmpty()) {
+      requirements.put(KubernetesConstants.CPU, Quantity.fromString(cpuLimit));
+    }
+
+    return requirements;
+  }
+
+  /**
+   * Configures the environment variables in the <code>container</code> with those Heron requires.
+   * Heron's values take precedence.
+   * @param container The <code>container</code> to be configured.
+   */
   @VisibleForTesting
   protected void configureContainerEnvVars(final V1Container container) {
     // Deduplicate on var name with Heron defaults take precedence.
@@ -681,6 +795,10 @@ protected void configureContainerEnvVars(final V1Container container) {
     );
   }
 
+  /**
+   * Generates a list of <code>Environment Variables</code> required by Heron to function.
+   * @return A list of configured <code>Environment Variables</code> required by Heron to function.
+   */
   @VisibleForTesting
   protected static List<V1EnvVar> getExecutorEnvVars() {
     final V1EnvVar envVarHost = new V1EnvVar();
@@ -698,6 +816,12 @@ protected static List<V1EnvVar> getExecutorEnvVars() {
     return Arrays.asList(envVarHost, envVarPodName);
   }
 
+  /**
+   * Configures the ports in the <code>container</code> with those Heron requires. Heron's values take precedence.
+   * @param remoteDebugEnabled Flag used to indicate if debugging ports need to be added.
+   * @param numberOfInstances The number of debugging ports to be opened.
+   * @param container <code>container</code> to be configured.
+   */
   @VisibleForTesting
   protected void configureContainerPorts(boolean remoteDebugEnabled, int numberOfInstances,
                                          final V1Container container) {
@@ -716,6 +840,10 @@ protected void configureContainerPorts(boolean remoteDebugEnabled, int numberOfI
     );
   }
 
+  /**
+   * Generates a list of <code>ports</code> required by Heron to function.
+   * @return A list of configured <code>ports</code> required by Heron to function.
+   */
   @VisibleForTesting
   protected static List<V1ContainerPort> getExecutorPorts() {
     List<V1ContainerPort> ports = new LinkedList<>();
@@ -728,6 +856,11 @@ protected static List<V1ContainerPort> getExecutorPorts() {
     return ports;
   }
 
+  /**
+   * Generate the debugging ports required by Heron.
+   * @param numberOfInstances The number of debugging ports to generate.
+   * @return A list of configured debugging <code>ports</code>.
+   */
   @VisibleForTesting
   protected static List<V1ContainerPort> getDebuggingPorts(int numberOfInstances) {
     List<V1ContainerPort> ports = new LinkedList<>();
@@ -742,6 +875,10 @@ protected static List<V1ContainerPort> getDebuggingPorts(int numberOfInstances)
     return ports;
   }
 
+  /**
+   * Adds volume mounts to the <code>container</code> that Heron requires. Heron's values taking precedence.
+   * @param container <code>container</code> to be configured.
+   */
   @VisibleForTesting
   protected void mountVolumeIfPresent(final V1Container container) {
     final Config config = getConfiguration();
@@ -789,9 +926,16 @@ public static double roundDecimal(double value, int places) {
     return Math.round(value * scale) / scale;
   }
 
+  /**
+   * Initiates the process of locating and loading <code>Pod Template</code> from a <code>ConfigMap</code>.
+   * The loaded text is then parsed into a usable <code>Pod Template</code>.
+   * @param isExecutor Flag to indicate loading of <code>Pod Template</code> for <code>Executor</code>
+   *                   or <code>Manager</code>.
+   * @return A <code>Pod Template</code> which is loaded and parsed from a <code>ConfigMap</code>.
+   */
   @VisibleForTesting
-  protected V1PodTemplateSpec loadPodFromTemplate() {
-    final Pair<String, String> podTemplateConfigMapName = getPodTemplateLocation();
+  protected V1PodTemplateSpec loadPodFromTemplate(boolean isExecutor) {
+    final Pair<String, String> podTemplateConfigMapName = getPodTemplateLocation(isExecutor);
 
     // Default Pod Template.
     if (podTemplateConfigMapName == null) {
@@ -838,10 +982,16 @@ protected V1PodTemplateSpec loadPodFromTemplate() {
     }
   }
 
+  /**
+   * Extracts the <code>ConfigMap</code> and <code>Pod Template</code> names from the CLI parameter.
+   * @param isExecutor Flag to indicate loading of <code>Pod Template</code> for <code>Executor</code>
+   *                   or <code>Manager</code>.
+   * @return A pair of the form <code>(ConfigMap, Pod Template)</code>.
+   */
   @VisibleForTesting
-  protected Pair<String, String> getPodTemplateLocation() {
+  protected Pair<String, String> getPodTemplateLocation(boolean isExecutor) {
     final String podTemplateConfigMapName = KubernetesContext
-        .getPodTemplateConfigMapName(getConfiguration());
+        .getPodTemplateConfigMapName(getConfiguration(), isExecutor);
 
     if (podTemplateConfigMapName == null) {
       return null;
@@ -864,6 +1014,11 @@ protected Pair<String, String> getPodTemplateLocation() {
     }
   }
 
+  /**
+   * Retrieves a <code>ConfigMap</code> from the K8s cluster in the API Server's namespace.
+   * @param configMapName Name of the <code>ConfigMap</code> to retrieve.
+   * @return The retrieved <code>ConfigMap</code>.
+   */
   @VisibleForTesting
   protected V1ConfigMap getConfigMap(String configMapName) {
     try {
@@ -996,12 +1151,14 @@ protected Pair<List<V1Volume>, List<V1VolumeMount>> createPersistentVolumeClaimV
    * Makes a call to generate <code>Volumes</code> and <code>Volume Mounts</code> and then inserts them.
    * @param podSpec All generated <code>V1Volume</code> will be placed in the <code>Pod Spec</code>.
    * @param executor All generated <code>V1VolumeMount</code> will be placed in the <code>Container</code>.
+   * @param configPVC <code>Persistent Volume Claim</code> configurations options.
    */
   @VisibleForTesting
   protected void configurePodWithPersistentVolumeClaimVolumesAndMounts(final V1PodSpec podSpec,
-                                                                       final V1Container executor) {
+      final V1Container executor,
+      final Map<String, Map<KubernetesConstants.VolumeClaimTemplateConfigKeys, String>> configPVC) {
     Pair<List<V1Volume>, List<V1VolumeMount>> volumesAndMounts =
-        createPersistentVolumeClaimVolumesAndMounts(persistentVolumeClaimConfigs);
+        createPersistentVolumeClaimVolumesAndMounts(configPVC);
 
     // Deduplicate on Names with Persistent Volume Claims taking precedence.
 
@@ -1072,7 +1229,7 @@ private void removePersistentVolumeClaims() {
   }
 
   /**
-   * Generates the <code>Label</code> which are attached to a Topologies Persistent Volume Claims.
+   * Generates the <code>Label</code> which are attached to a Topology's Persistent Volume Claims.
    * @param topologyName Attached to the topology match label.
    * @return A map consisting of the <code>label-value</code> pairs to be used in <code>Label</code>s.
    */
@@ -1086,6 +1243,17 @@ protected static Map<String, String> getPersistentVolumeClaimLabels(String topol
     };
   }
 
+  /**
+   * Generates the <code>StatefulSet</code> name depending on if it is a <code>Executor</code> or
+   * <code>Manager</code>.
+   * @param isExecutor Flag used to generate name for <code>Executor</code> or <code>Manager</code>.
+   * @return String <code>"topology-name"-executors</code>.
+   */
+  private String getStatefulSetName(boolean isExecutor) {
+    return String.format("%s-%s", getTopologyName(),
+        isExecutor ? KubernetesConstants.EXECUTOR_NAME + "s" : KubernetesConstants.MANAGER_NAME);
+  }
+
   /**
    * Generates the <code>Selector</code> match labels with which resources in this topology can be found.
    * @return A label of the form <code>app=heron,topology=topology-name</code>.
diff --git a/website2/docs/schedulers-k8s-execution-environment.md b/website2/docs/schedulers-k8s-execution-environment.md
new file mode 100644
index 00000000000..cbc195641de
--- /dev/null
+++ b/website2/docs/schedulers-k8s-execution-environment.md
@@ -0,0 +1,527 @@
+---
+id: schedulers-k8s-execution-environment
+title: Kubernetes Execution Environment Customization
+sidebar_label:  Kubernetes Execution Environment Customization
+---
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+      http://www.apache.org/licenses/LICENSE-2.0
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+# Customizing the Heron Execution Environment
+
+This document demonstrates how you can customize various aspects of the Heron execution environment when using the Kubernetes Scheduler.
+
+<br>
+
+***Table of contents:***
+- [Customizing the Heron Execution Environment](#customizing-the-heron-execution-environment)
+  - [Customizing a Topology's Execution Environment Using Pod Templates](#customizing-a-topologys-execution-environment-using-pod-templates)
+    - [Preparation](#preparation)
+      - [Pod Templates](#pod-templates)
+      - [Configuration Maps](#configuration-maps)
+    - [Submitting](#submitting)
+    - [Heron Configured Items in Pod Templates](#heron-configured-items-in-pod-templates)
+      - [Executor and Manager Containers](#executor-and-manager-containers)
+      - [Pod](#pod)
+  - [Adding Persistent Volumes via the Command Line Interface](#adding-persistent-volumes-via-the-command-line-interface)
+    - [Usage](#usage)
+      - [Example](#example)
+    - [Submitting](#submitting-1)
+    - [Required and Optional Configuration Items](#required-and-optional-configuration-items)
+    - [Configuration Items Created and Entries Made](#configuration-items-created-and-entries-made)
+  - [Setting Limits and Requests via the Command Line Interface](#setting-limits-and-requests-via-the-command-line-interface)
+    - [Usage](#usage-1)
+      - [Example](#example-1)
+
+<br>
+
+---
+
+<br>
+
+## Customizing a Topology's Execution Environment Using Pod Templates
+
+<br>
+
+> This section demonstrates how you can utilize custom [Pod Templates](https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates) embedded in [Configuration Maps](https://kubernetes.io/docs/concepts/configuration/configmap/) for your Topology's `Executor`s and `Manager` (hereinafter referred to as `Heron containers`). You may specify different Pod Templates for different topologies.
+
+<br/>
+
+When you deploy a topology to Heron on Kubernetes, you may specify individual Pod Templates to be used in your topology's `Executor`s and `Manager`. This can be achieved by providing valid Pod Templates, and embedding the Pod Templates in Configuration Maps. By default, Heron will use a minimally configured Pod Template which is adequate to deploy a topology.
+
+Pod Templates will allow you to configure most aspects of your topology's execution environment, with some exceptions. There are some aspects of Pods for which Heron will have the final say, and which will not be user-customizable. Please view the [tables](#heron-configured-items-in-pod-templates) at the end of this section to identify what is set by Heron.
+
+<br>
+
+> ***System Administrators:***
+>
+> * You may wish to disable the ability to load custom Pod Templates. To achieve this, you must pass the define option `-D heron.kubernetes.pod.template.disabled=true` to the Heron API Server on the command line when launching. This command has been added to the Kubernetes configuration files to deploy the Heron API Server and is set to `false` by default.
+> * If you have a custom `Role` for the Heron API Server you will need to ensure the `ServiceAccount` attached to the API server, via a `RoleBinding`, has the correct permissions to access the `ConfigMaps`:
+>
+>```yaml
+>rules:
+>- apiGroups: 
+>  - ""
+>  resources: 
+>  - configmaps
+>  verbs: 
+>  - get
+>  - list
+>```
+
+<br>
+
+### Preparation
+
+To deploy a custom Pod Template to Kubernetes with your topology, you must provide a valid Pod Template embedded in a valid Configuration Map. We will be using the following variables throughout this document, some of which are reserved variable names:
+
+* `POD-TEMPLATE-NAME`: This is the name of the Pod Template's YAML definition file. This is ***not*** a reserved variable and is a place-holder name.
+* `CONFIG-MAP-NAME`: This is the name that will be used by the Configuration Map in which the Pod Template will be embedded by `kubectl`. This is ***not*** a reserved variable and is a place-holder name.
+* `heron.kubernetes.[executor | manager].pod.template`: This variable name is used as the key passed to Heron for the `--config-property` on the CLI. This ***is*** a reserved variable name.
+
+***NOTE***: Please do ***not*** use the `.` (period character) in the name of the `CONFIG-MAP-NAME`. This character will be used as a delimiter when submitting your topologies.
+
+It is highly advised that you validate your Pod Templates before placing them in a `ConfigMap` to isolate any validity issues using a tool such as [Kubeval](https://kubeval.instrumenta.dev/) or the built-in `dry-run` functionality in Kubernetes. Whilst these tools are handy, they will not catch all potential errors in Kubernetes configurations.
+
+***NOTE***: When submitting a Pod Template to customize an `Executor` or `Manager`, Heron will look for containers named `executor` and `manager` respectively. These containers will be modified to support the functioning of Heron, please read further below.
+
+#### Pod Templates
+
+An example of the Pod Template format is provided below, and is derived from the configuration for the Heron Tracker Pod:
+
+```yaml
+apiVersion: v1
+kind: PodTemplate
+metadata:
+  name: heron-tracker
+  namespace: default
+template:
+  metadata:
+    labels:
+      app: heron-tracker
+  spec:
+    containers:
+      - name: heron-tracker
+        image: apache/heron:latest
+        ports:
+          - containerPort: 8888
+            name: api-port
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "200M"
+          limits:
+            cpu: "400m"
+            memory: "512M"
+```
+
+You would need to save this file as `POD-TEMPLATE-NAME`. Once you have a valid Pod Template you may proceed to generate a `ConfigMap`.
+
+#### Configuration Maps
+
+> You must place the `ConfigMap` in the same namespace as the Heron API Server using the `--namespace` option in the commands below if the API Server is not in the `default` namespace.
+
+To generate a `ConfigMap` you will need to run the following command:
+
+```bash
+kubectl create configmap CONFIG-MAP-NAME --from-file path/to/POD-TEMPLATE-NAME
+```
+
+You may then want to verify the contents of the `ConfigMap` by running the following command:
+
+```bash
+kubectl get configmaps CONFIG-MAP-NAME -o yaml
+```
+
+The `ConfigMap` should appear similar to the one below for our example:
+
+```yaml
+apiVersion: v1
+data:
+  POD-TEMPLATE-NAME: |
+    apiVersion: v1
+    kind: PodTemplate
+    metadata:
+      name: heron-tracker
+      namespace: default
+    template:
+      metadata:
+        labels:
+          app: heron-tracker
+      spec:
+        containers:
+          - name: heron-tracker
+            image: apache/heron:latest
+            ports:
+              - containerPort: 8888
+                name: api-port
+            resources:
+              requests:
+                cpu: "100m"
+                memory: "200M"
+              limits:
+                cpu: "400m"
+                memory: "512M"
+kind: ConfigMap
+metadata:
+  creationTimestamp: "2021-09-27T21:55:30Z"
+  name: CONFIG-MAP-NAME
+  namespace: default
+  resourceVersion: "1313"
+  uid: ba001653-03d9-4ac8-804c-d2c55c974281
+```
+
+### Submitting
+
+To use the `ConfigMap` for a topology you would will need to submit with the additional flag `--confg-property`. The `--config-property key=value` takes a key-value pair:
+
+* Key: `heron.kubernetes.[executor | manager].pod.template`
+* Value: `CONFIG-MAP-NAME.POD-TEMPLATE-NAME`
+
+Please note that you must concatenate `CONFIG-MAP-NAME` and `POD-TEMPLATE-NAME` with a **`.`** (period character).
+
+For example:
+
+```bash
+heron submit kubernetes \
+  --service-url=http://localhost:8001/api/v1/namespaces/default/services/heron-apiserver:9000/proxy \
+  ~/.heron/examples/heron-api-examples.jar \
+  org.apache.heron.examples.api.AckingTopology acking \
+  --config-property heron.kubernetes.executor.pod.template=CONFIG-MAP-NAME.POD-TEMPLATE-NAME \
+  --config-property heron.kubernetes.manager.pod.template=CONFIG-MAP-NAME.POD-TEMPLATE-NAME
+```
+
+### Heron Configured Items in Pod Templates
+
+Heron will locate the containers named `executor` and/or `manager` in the Pod Template and customize them as outlined below. All other containers within the Pod Templates will remain unchanged.
+
+#### Executor and Manager Containers
+
+All metadata for the `Heron containers` will be overwritten by Heron. In some other cases, values from the Pod Template for the `executor` and `manager` will be overwritten by Heron as outlined below.
+
+| Name | Description | Policy |
+|---|---|---|
+| `image` | The `Heron container`'s image. | Overwritten by Heron using values from the config.
+| `env` | Environment variables are made available within the container. The `HOST` and `POD_NAME` keys are required by Heron and are thus reserved. | Merged with Heron's values taking precedence. Deduplication is based on `name`.
+| `ports` | Port numbers opened within the container. Some of these port numbers are required by Heron and are thus reserved. The reserved ports are defined in Heron's constants as [`6001`-`6010`]. | Merged with Heron's values taking precedence. Deduplication is based on the `containerPort` value.
+| `limits` <br> `requests` | Heron will attempt to load values for `cpu` and `memory` from configs. | Heron's values take precedence over those in the Pod Templates.
+| `volumeMounts` | These are the mount points within the `Heron container` for the `volumes` available in the Pod. | Merged with Heron's values taking precedence. Deduplication is based on the `name` value.
+| Annotation: `prometheus.io/scrape` | Flag to indicate whether Prometheus logs can be scraped and is set to `true`. | Value is overridden by Heron. |
+| Annotation `prometheus.io/port` | Port address for Prometheus log scraping and is set to `8080`. | Values are overridden by Heron.
+| Annotation: Pod | Pod's revision/version hash. | Automatically set.
+| Annotation: Service | Labels services can use to attach to the Pod. | Automatically set.
+| Label: `app` | Name of the application launching the Pod and is set to `Heron`. | Values are overridden by Heron.
+| Label: `topology`| The name of topology which was provided when submitting. | User-defined and supplied on the CLI.
+
+#### Pod
+
+The following items will be set in the Pod Template's `spec` by Heron.
+
+| Name | Description | Policy |
+|---|---|---|
+`terminationGracePeriodSeconds` | Grace period to wait before shutting down the Pod after a `SIGTERM` signal and is set to `0` seconds. | Values are overridden by Heron.
+| `tolerations` | Attempts to schedule Pods with `taints` onto nodes hosting Pods with matching `taints`. The entries below are included by default. <br>  Keys:<br>`node.kubernetes.io/not-ready` <br> `node.kubernetes.io/unreachable` <br> Values (common):<br> `operator: Exists`<br> `effect: NoExecute`<br> `tolerationSeconds: 10L` | Merged with Heron's values taking precedence. Deduplication is based on the `key` value.
+| `containers` | Configurations for containers to be launched within the Pod. | All containers, excluding the `Heron container`s, are loaded as-is.
+| `volumes` | Volumes to be made available to the entire Pod. | Merged with Heron's values taking precedence. Deduplication is based on the `name` value.
+| `secretVolumes` | Secrets to be mounted as volumes within the Pod. | Loaded from the Heron configs if present.
+
+<br>
+
+---
+<br>
+
+## Adding Persistent Volumes via the Command Line Interface
+
+<br>
+
+> This section demonstrates how you can utilize both static and dynamically backed [Persistent Volume Claims](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) in the `Executor` and `Manager` containers (hereinafter referred to as `Heron containers`). You will need to enable Dynamic Provisioning in your Kubernetes cluster to proceed to use the dynamic provisioning functionality.
+
+<br/>
+
+It is possible to leverage Persistent Volumes with custom Pod Templates but the Volumes you add will be shared between all  `Executor` Pods in the topology when customizing the `Executor`s.
+
+The CLI commands allow you to configure a Persistent Volume Claim (dynamically or statically backed) which will be unique and isolated to each Pod and mounted in a single `Heron container` when you submit your topology with a claim name of `OnDemand`. Using any claim name other than on `OnDemand` will permit you to configure a shared Persistent Volume without a custom Pod Template which will be shared between all `Executor` Pods when customizing them. The CLI commands override any configurations you may have present in the Pod Template, but Heron's configurations will take precedence over all others.
+
+Some use cases include process checkpointing, caching of results for later use in the process, intermediate results which could prove useful in analysis (ETL/ELT to a data lake or warehouse), as a source of data enrichment, etc.
+
+**Note:** Heron ***will*** remove any dynamically backed Persistent Volume Claims it creates when a topology is terminated. Please be aware that Heron uses the following `Labels` to locate the claims it has created:
+```yaml
+metadata:
+  labels:
+    topology: <topology-name>
+    onDemand: true
+```
+
+<br>
+
+> ***System Administrators:***
+>
+> * You may wish to disable the ability to configure Persistent Volume Claims specified via the CLI. To achieve this, you must pass the define option `-D heron.kubernetes.persistent.volume.claims.cli.disabled=true`to the Heron API Server on the command line when launching. This command has been added to the Kubernetes configuration files to deploy the Heron API Server and is set to `false` by default.
+> * If you have a custom `Role`/`ClusterRole` for the Heron API Server you will need to ensure the `ServiceAccount` attached to the API server has the correct permissions to access the `Persistent Volume Claim`s:
+>
+>```yaml
+>rules:
+>- apiGroups: 
+>  - ""
+>  resources: 
+>  - persistentvolumeclaims
+>  verbs: 
+>  - create
+>  - delete
+>  - get
+>  - list
+>  - deletecollection
+>```
+
+<br>
+
+### Usage
+
+To configure a Persistent Volume Claim you must use the `--config-property` option with the `heron.kubernetes.[executor | manager].volumes.persistentVolumeClaim.` command prefix. Heron will not validate your Persistent Volume Claim configurations, so please validate them to ensure they are well-formed. All names must comply with the [*lowercase RFC-1123*](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/) standard.
+
+The command pattern is as follows:
+`heron.kubernetes.[executor | manager].volumes.persistentVolumeClaim.[VOLUME NAME].[OPTION]=[VALUE]`
+
+The currently supported CLI `options` are:
+
+* `claimName`
+* `storageClass`
+* `sizeLimit`
+* `accessModes`
+* `volumeMode`
+* `path`
+* `subPath`
+
+***Note:*** A `claimName` of `OnDemand` will create unique Volumes for each `Heron container` as well as deploy a Persistent Volume Claim for each Volume. Any other claim name will result in a shared Volume being created between all Pods in the topology.
+
+***Note:*** The `accessModes` must be a comma-separated list of values *without* any white space. Valid values can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes).
+
+***Note:*** If a `storageClassName` is specified and there are no matching Persistent Volumes then [dynamic provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) must be enabled. Kubernetes will attempt to locate a Persistent Volume that matches the `storageClassName` before it attempts to use dynamic provisioning. If a `storageClassName` is not specified there must be [Persistent Volumes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/) provisioned manually with the `storageClassName` of `standard`.
+
+<br>
+
+#### Example
+
+A series of example commands to add `Persistent Volumes` to `Executor`s, and the `YAML` entries they make in their respective configurations, are as follows.
+
+***Dynamic:***
+
+```bash
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.claimName=OnDemand
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.storageClassName=storage-class-name-of-choice
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=comma,separated,list
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=volume-mode-of-choice
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.path=/path/to/mount
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.subPath=/sub/path/to/mount
+```
+
+Generated `Persistent Volume Claim`:
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  labels:
+    app: heron
+    onDemand: "true"
+    topology: <topology-name>
+  name: volumenameofchoice-<topology-name>-[Ordinal]
+spec:
+  accessModes:
+  - comma
+  - separated
+  - list
+  resources:
+    requests:
+      storage: 555Gi
+  storageClassName: storage-class-name-of-choice
+  volumeMode: volume-mode-of-choice
+```
+
+Pod Spec entries for `Volume`:
+
+```yaml
+volumes:
+  - name: volumenameofchoice
+    persistentVolumeClaim:
+      claimName: volumenameofchoice-<topology-name>-[Ordinal]
+```
+
+`Executor` container entries for `Volume Mounts`:
+
+```yaml
+volumeMounts:
+  - mountPath: /path/to/mount
+    subPath: /sub/path/to/mount
+    name: volumenameofchoice
+```
+
+<br>
+
+***Static:***
+
+```bash
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.claimName=OnDemand
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=comma,separated,list
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=volume-mode-of-choice
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.path=/path/to/mount
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.subPath=/sub/path/to/mount
+```
+
+Generated `Persistent Volume Claim`:
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  labels:
+    app: heron
+    onDemand: "true"
+    topology: <topology-name>
+  name: volumenameofchoice-<topology-name>-[Ordinal]
+spec:
+  accessModes:
+  - comma
+  - separated
+  - list
+  resources:
+    requests:
+      storage: 555Gi
+  storageClassName: standard
+  volumeMode: volume-mode-of-choice
+```
+
+Pod Spec entries for `Volume`:
+
+```yaml
+volumes:
+  - name: volumenameofchoice
+    persistentVolumeClaim:
+      claimName: volumenameofchoice-<topology-name>-[Ordinal]
+```
+
+`Executor` container entries for `Volume Mounts`:
+
+```yaml
+volumeMounts:
+  - mountPath: /path/to/mount
+    subPath: /sub/path/to/mount
+    name: volumenameofchoice
+```
+
+<br>
+
+### Submitting
+
+A series of example commands to sumbit a topology using the *dynamic* example CLI commands above:
+
+```bash
+heron submit kubernetes \
+  --service-url=http://localhost:8001/api/v1/namespaces/default/services/heron-apiserver:9000/proxy \
+  ~/.heron/examples/heron-api-examples.jar \
+  org.apache.heron.examples.api.AckingTopology acking \
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.claimName=OnDemand \
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.storageClassName=storage-class-name-of-choice \
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=comma,separated,list \
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi \
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=volume-mode-of-choice \
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.path=/path/to/mount \
+--config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.volumenameofchoice.subPath=/sub/path/to/mount
+```
+
+### Required and Optional Configuration Items
+
+The following table outlines CLI options which are either ***required*** ( &#x2705; ), ***optional*** ( &#x2754; ), or ***not available*** ( &#x274c; ) depending on if you are using dynamically/statically backed or shared `Volume`s.
+
+| Option | Dynamic | Static | Shared
+|---|---|---|---|
+| `VOLUME NAME` | &#x2705; | &#x2705; | &#x2705;
+| `claimName` | `OnDemand` | `OnDemand` | A valid name
+| `path` | &#x2705; | &#x2705; | &#x2705;
+| `subPath` | &#x2754; | &#x2754; | &#x2754;
+| `storageClassName` | &#x2705; | &#x274c; | &#x274c;
+| `accessModes` | &#x2705; | &#x2705; | &#x274c;
+| `sizeLimit` | &#x2754; | &#x2754; | &#x274c;
+| `volumeMode` | &#x2754; | &#x2754; | &#x274c;
+
+<br>
+
+***Note:*** The `VOLUME NAME` will be extracted from the CLI command and a `claimName` is a always required.
+
+<br>
+
+### Configuration Items Created and Entries Made
+
+The configuration items and entries in the tables below will made in their respective areas.
+
+A `Volume` and a `Volume Mount` will be created for each `volume name` which you specify. Additionally, one `Persistent Volume Claim` will be created for each `Volume` specified as dynamic using the `OnDemand` claim name.
+
+| Name | Description | Policy |
+|---|---|---|
+| `VOLUME NAME` | The `name` of the `Volume`. | Entries made in the `Persistent Volume Claim`'s spec, the Pod Spec's `Volumes`, and the `Heron containers` `volumeMounts`.
+| `claimName` | A Claim name for the Persistent Volume. | If `OnDemand` is provided as the parameter then a unique Volume and Persistent Volume Claim will be created. Any other name will result in a shared Volume between all Pods in the topology with only a Volume and Volume Mount being added.
+| `path` | The `mountPath` of the `Volume`. | Entries made in the `Heron containers` `volumeMounts`.
+| `subPath` | The `subPath` of the `Volume`. | Entries made in the `Heron containers` `volumeMounts`.
+| `storageClassName` | The identifier name used to reference the dynamic `StorageClass`. | Entries made in the `Persistent Volume Claim` and Pod Spec's `Volume`.
+| `accessModes` | A comma-separated list of [access modes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes). | Entries made in the `Persistent Volume Claim`.
+| `sizeLimit` | A resource request for storage space [units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory). | Entries made in the `Persistent Volume Claim`.
+| `volumeMode` | Either `FileSystem` (default) or `Block` (raw block). [Read more](https://kubernetes.io/docs/concepts/storage/_print/#volume-mode). | Entries made in the `Persistent Volume Claim`.
+| Labels | Two labels for `topology` and `onDemand` provisioning are added. | These labels are only added to dynamically backed `Persistent Volume Claim`s created by Heron to support the removal of any claims created when a topology is terminated.
+
+<br>
+
+---
+
+<br>
+
+## Setting Limits and Requests via the Command Line Interface
+
+> This section demonstrates how you can configure a topology's `Executor` and/or `Manager` (hereinafter referred to as `Heron containers`) resource `Requests` and `Limits` through CLI commands.
+
+<br/>
+
+You may configure an individual topology's `Heron container`'s resource `Requests` and `Limits` during submission through CLI commands. The default behaviour is to acquire values for resources from Configurations and for them to be common between the `Executor`s and the `Manager` for a topology.
+
+<br>
+
+### Usage
+
+The command pattern is as follows:
+`heron.kubernetes.[executor | manager].[limits | requests].[OPTION]=[VALUE]`
+
+The currently supported CLI `options` and their associated `values` are:
+
+* `cpu`: A natural number indicating the number of [CPU units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu).
+* `memory`: A natural number indicating the amount of [memory units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory).
+
+<br>
+
+#### Example
+
+An example submission command is as follows.
+
+***Limits and Requests:***
+
+```bash
+~/bin/heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \
+org.apache.heron.examples.api.AckingTopology acking \
+--config-property heron.kubernetes.manager.limits.cpu=2 \
+--config-property heron.kubernetes.manager.limits.memory=3 \
+--config-property heron.kubernetes.manager.requests.cpu=1 \
+--config-property heron.kubernetes.manager.requests.memory=2
+```
diff --git a/website2/docs/schedulers-k8s-persistent-volume-claims.md b/website2/docs/schedulers-k8s-persistent-volume-claims.md
deleted file mode 100644
index 994c22c05bd..00000000000
--- a/website2/docs/schedulers-k8s-persistent-volume-claims.md
+++ /dev/null
@@ -1,257 +0,0 @@
----
-id: schedulers-k8s-persistent-volume-claims
-title: Kubernetes Persistent Volume Claims via CLI
-sidebar_label: Kubernetes Persistent Volume Claims (CLI)
----
-<!--
-    Licensed to the Apache Software Foundation (ASF) under one
-    or more contributor license agreements.  See the NOTICE file
-    distributed with this work for additional information
-    regarding copyright ownership.  The ASF licenses this file
-    to you under the Apache License, Version 2.0 (the
-    "License"); you may not use this file except in compliance
-    with the License.  You may obtain a copy of the License at
-      http://www.apache.org/licenses/LICENSE-2.0
-    Unless required by applicable law or agreed to in writing,
-    software distributed under the License is distributed on an
-    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-    KIND, either express or implied.  See the License for the
-    specific language governing permissions and limitations
-    under the License.
--->
-
-> This document demonstrates how you can utilize both static and dynamically backed [Persistent Volume Claims](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) in the `Executor` containers. You will need to enable Dynamic Provisioning in your Kubernetes cluster to proceed to use the dynamic provisioning functionality.
-
-<br/>
-
-It is possible to leverage Persistent Volumes with custom Pod Templates but the Volumes you add will be shared between all Pods in the topology.
-
-The CLI commands allow you to configure a Persistent Volume Claim (dynamically or statically backed) which will be unique and isolated to each Pod and mounted in a single `Executor` when you submit your topology with a Claim name of `OnDemand`. Using any Claim name other than on `OnDemand` will permit you to configure a shared Persistent Volume without a custom Pod Template which will be specific to an individual Pod. The CLI commands override any configurations you may have present in the Pod Template, but Heron's configurations will take precedence over all others.
-
-Some use cases include process checkpointing, caching of results for later use in the process, intermediate results which could prove useful in analysis (ETL/ELT to a data lake or warehouse), as a source of data enrichment, etc.
-
-**Note:** Heron ***will*** remove any dynamically backed Persistent Volume Claims it creates when a topology is terminated. Please be aware that Heron uses the following `Labels` to locate the claims it has created:
-```yaml
-metadata:
-  labels:
-    topology: <topology-name>
-    onDemand: true
-```
-
-<br>
-
-> ***System Administrators:***
->
-> * You may wish to disable the ability to configure dynamic Persistent Volume Claims specified on the CLI. To achieve this, you must pass the define option `-D heron.kubernetes.persistent.volume.claims.cli.disabled=true` to the Heron API Server on the command line during boot. This command has been added to the Kubernetes configuration files to deploy the Heron API Server and is set to `false` by default.
-> * If you have a custom `Role`/`ClusterRole` for the Heron API Server you will need to ensure the `ServiceAccount` attached to the API server has the correct permissions to access the `Persistent Volume Claim`s:
->
->```yaml
->rules:
->- apiGroups: 
->  - ""
->  resources: 
->  - persistentvolumeclaims
->  verbs: 
->  - create
->  - delete
->  - get
->  - list
->  - deletecollection
->```
-
-<br>
-
-## Usage
-
-To configure a Persistent Volume Claim you must use the `--config-property` option with the `heron.kubernetes.volumes.persistentVolumeClaim.` command prefix. Heron will not validate your Persistent Volume Claim configurations, so please validate them to ensure they are well-formed. All names must comply with the [*lowercase RFC-1123*](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/) standard.
-
-The command pattern is as follows:
-`heron.kubernetes.volumes.persistentVolumeClaim.[VOLUME NAME].[OPTION]=[VALUE]`
-
-The currently supported CLI `options` are:
-
-* `claimName`
-* `storageClass`
-* `sizeLimit`
-* `accessModes`
-* `volumeMode`
-* `path`
-* `subPath`
-
-***Note:*** A `claimName` of `OnDemand` will create unique Volumes for each `Executor` as well as deploy a Persistent Volume Claim for each Volume. Any other Claim name will result in a shared Volume being created between all Pods in the topology.
-
-***Note:*** The `accessModes` must be a comma separated list of values *without* any white space. Valid values can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes).
-
-***Note:*** If a `storageClassName` is specified and there are no matching Persistent Volumes then [dynamic provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) must be enabled. Kubernetes will attempt to locate a Persistent Volume that matches the `storageClassName` before it attempts to use dynamic provisioning. If a `storageClassName` is not specified there must be [Persistent Volumes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/) provisioned manually with the `storageClassName` of `standard`.
-
-<br>
-
-### Example
-
-An example series of commands and the `YAML` entries they make in their respective configurations are as follows.
-
-***Dynamic:***
-
-```bash
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.claimName=OnDemand
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.storageClassName=storage-class-name-of-choice
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=comma,separated,list
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=volume-mode-of-choice
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.path=/path/to/mount
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.subPath=/sub/path/to/mount
-```
-
-Generated `Persistent Volume Claim`:
-
-```yaml
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  labels:
-    app: heron
-    onDemand: "true"
-    topology: <topology-name>
-  name: volumenameofchoice-<topology-name>-[Ordinal]
-spec:
-  accessModes:
-  - comma
-  - separated
-  - list
-  resources:
-    requests:
-      storage: 555Gi
-  storageClassName: storage-class-name-of-choice
-  volumeMode: volume-mode-of-choice
-```
-
-Pod Spec entries for `Volume`:
-
-```yaml
-volumes:
-  - name: volumenameofchoice
-    persistentVolumeClaim:
-      claimName: volumenameofchoice-<topology-name>-[Ordinal]
-```
-
-`Executor` container entries for `Volume Mounts`:
-
-```yaml
-volumeMounts:
-  - mountPath: /path/to/mount
-    subPath: /sub/path/to/mount
-    name: volumenameofchoice
-```
-
-<br>
-
-***Static:***
-
-```bash
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.claimName=OnDemand
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=comma,separated,list
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=volume-mode-of-choice
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.path=/path/to/mount
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.subPath=/sub/path/to/mount
-```
-
-Generated `Persistent Volume Claim`:
-
-```yaml
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  labels:
-    app: heron
-    onDemand: "true"
-    topology: <topology-name>
-  name: volumenameofchoice-<topology-name>-[Ordinal]
-spec:
-  accessModes:
-  - comma
-  - separated
-  - list
-  resources:
-    requests:
-      storage: 555Gi
-  storageClassName: standard
-  volumeMode: volume-mode-of-choice
-```
-
-Pod Spec entries for `Volume`:
-
-```yaml
-volumes:
-  - name: volumenameofchoice
-    persistentVolumeClaim:
-      claimName: volumenameofchoice-<topology-name>-[Ordinal]
-```
-
-`Executor` container entries for `Volume Mounts`:
-
-```yaml
-volumeMounts:
-  - mountPath: /path/to/mount
-    subPath: /sub/path/to/mount
-    name: volumenameofchoice
-```
-
-<br>
-
-## Submitting
-
-An example of sumbitting a topology using the *dynamic* example CLI commands above:
-
-```bash
-heron submit kubernetes \
-  --service-url=http://localhost:8001/api/v1/namespaces/default/services/heron-apiserver:9000/proxy \
-  ~/.heron/examples/heron-api-examples.jar \
-  org.apache.heron.examples.api.AckingTopology acking \
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.claimName=OnDemand \
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.storageClassName=storage-class-name-of-choice \
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.accessModes=comma,separated,list \
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.sizeLimit=555Gi \
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.volumeMode=volume-mode-of-choice \
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.path=/path/to/mount \
---config-property heron.kubernetes.volumes.persistentVolumeClaim.volumenameofchoice.subPath=/sub/path/to/mount
-```
-
-## Required and Optional Configuration Items
-
-The following table outlines CLI options which are either ***required*** ( &#x2705; ), ***optional*** ( &#x2754; ), or ***not available*** ( &#x274c; ) depending on if you are using dynamic/statically backed or shared `Volume`.
-
-| Option | Dynamic | Static | Shared
-|---|---|---|---|
-| `VOLUME NAME` | &#x2705; | &#x2705; | &#x2705;
-| `claimName` | `OnDemand` | `OnDemand` | A valid name
-| `path` | &#x2705; | &#x2705; | &#x2705;
-| `subPath` | &#x2754; | &#x2754; | &#x2754;
-| `storageClassName` | &#x2705; | &#x274c; | &#x274c;
-| `accessModes` | &#x2705; | &#x2705; | &#x274c;
-| `sizeLimit` | &#x2754; | &#x2754; | &#x274c;
-| `volumeMode` | &#x2754; | &#x2754; | &#x274c;
-
-<br>
-
-***Note:*** The `VOLUME NAME` will be extracted from the CLI command and a `claimName` is a always required.
-
-<br>
-
-## Configuration Items Created and Entries Made
-
-The configuration items and entries in the tables below will made in their respective areas.
-
-One `Persistent Volume Claim`, a `Volume`, and a `Volume Mount` will be created for each `volume name` which you specify. Each will be unique to a Pod within the topology.
-
-| Name | Description | Policy |
-|---|---|---|
-| `VOLUME NAME` | The `name` of the `Volume`. | Entries made in the `Persistent Volume Claim`'s spec, the Pod Spec's `Volumes`, and the `executor` containers `volumeMounts`.
-| `claimName` | A Claim name for the Persistent Volume. | If `OnDemand` is provided as the parameter then a unique Volume and Persistent Volume Claim will be created. Any other name will result in a shared Volume between all Pods in the topology with only a Volume and Volume Mount being added.
-| `path` | The `mountPath` of the `Volume`. | Entries made in the `executor` containers `volumeMounts`.
-| `subPath` | The `subPath` of the `Volume`. | Entries made in the `executor` containers `volumeMounts`.
-| `storageClassName` | The identifier name used to reference the dynamic `StorageClass`. | Entries made in the `Persistent Volume Claim` and Pod Spec's `Volume`.
-| `accessModes` | A comma separated list of access modes. | Entries made in the `Persistent Volume Claim`.
-| `sizeLimit` | A resource request for storage space. | Entries made in the `Persistent Volume Claim`.
-| `volumeMode` | Either `FileSystem` (default) or `Block` (raw block). [Read more](https://kubernetes.io/docs/concepts/storage/_print/#volume-mode). | Entries made in the `Persistent Volume Claim`.
-| Labels | Two labels for `topology` and `onDemand` provisioning are added. | These labels are only added to dynamically backed `Persistent Volume Claim`'s created by Heron to support the removal of any claims created when a topology is terminated.
diff --git a/website2/docs/schedulers-k8s-pod-templates.md b/website2/docs/schedulers-k8s-pod-templates.md
deleted file mode 100644
index e6fe3c981eb..00000000000
--- a/website2/docs/schedulers-k8s-pod-templates.md
+++ /dev/null
@@ -1,201 +0,0 @@
----
-id: schedulers-k8s-pod-templates
-title: Kubernetes Pod Templates
-sidebar_label:  Kubernetes Pod Templates
----
-<!--
-    Licensed to the Apache Software Foundation (ASF) under one
-    or more contributor license agreements.  See the NOTICE file
-    distributed with this work for additional information
-    regarding copyright ownership.  The ASF licenses this file
-    to you under the Apache License, Version 2.0 (the
-    "License"); you may not use this file except in compliance
-    with the License.  You may obtain a copy of the License at
-      http://www.apache.org/licenses/LICENSE-2.0
-    Unless required by applicable law or agreed to in writing,
-    software distributed under the License is distributed on an
-    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-    KIND, either express or implied.  See the License for the
-    specific language governing permissions and limitations
-    under the License.
--->
-
-> This document demonstrates how you can utilize custom [Pod Templates](https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates) embedded in [Configuration Maps](https://kubernetes.io/docs/concepts/configuration/configmap/) for your computation nodes - i.e., Spouts and Bolts. You may specify different Pod Templates for different topologies.
-
-<br/>
-
-When you deploy a topology to Heron on Kubernetes, you may specify a Pod Template to be used on the computation nodes. This can be achieved by providing a valid Pod Template, and embedding the Pod Template within a Configuration Map. By default, Heron will use a minimally configured Pod Template which is adequate to deploy a topology.
-
-Pod Templates will allow you to configure most aspects of the Pods where the computations occur, with some exceptions. There are some aspects of Pods for which Heron will have the final say, and which will not be user-customizable. Please view the tables at the end of this document to identify what is set by Heron.
-
-<br>
-
-> ***System Administrators:***
->
-> * You may wish to disable the ability to load custom Pod Templates. To achieve this, you must pass the define option `-D heron.kubernetes.pod.template.configmap.disabled=true` to the Heron API Server on the command line during boot. This command has been added to the Kubernetes configuration files to deploy the Heron API Server and is set to `false` by default.
-> * If you have a custom `Role`/`ClusterRole` for the Heron API Server you will need to ensure the `ServiceAccount` attached to the API server has the correct permissions to access the `ConfigMaps`:
->
->```yaml
->rules:
->- apiGroups: 
->  - ""
->  resources: 
->  - configmaps
->  verbs: 
->  - get
->  - watch
->  - list
->```
-
-<br>
-
-## Preparation
-
-To deploy a custom Pod Template to Kubernetes with your topology, you must provide a valid Pod Template embedded in a valid Configuration Map. We will be using the following variables throughout this document, some of which are reserved variable names:
-
-* `POD-TEMPLATE-NAME`: This is the name of the Pod Template's YAML definition file. This is ***not*** a reserved variable and is a place-holder name.
-* `CONFIG-MAP-NAME`: This is the name which will be used by the Configuration Map in which the Pod Template will be embedded by `kubectl`. This is ***not*** a reserved variable and is a place-holder name.
-* `heron.kubernetes.pod.template.configmap.name`: This variable name used as the key passed to Heron for the `--config-property` on the CLI. This ***is*** a reserved variable name.
-
-***NOTE***: Please do ***not*** use the `.` (period character) in the name of the `CONFIG-MAP-NAME`. This character will be used as a delimiter when submitting your topologies.
-
-It is highly advised that you validate your Pod Templates before placing them in a `ConfigMap` to isolate any validity issues using a tool such as [Kubeval](https://kubeval.instrumenta.dev/).
-
-### Pod Templates
-
-An example of a Pod Template is provided below, and is derived from the configuration for the Heron Tracker Pod:
-
-```yaml
-apiVersion: v1
-kind: PodTemplate
-metadata:
-  name: heron-tracker
-  namespace: default
-template:
-  metadata:
-    labels:
-      app: heron-tracker
-  spec:
-    containers:
-      - name: heron-tracker
-        image: apache/heron:latest
-        ports:
-          - containerPort: 8888
-            name: api-port
-        resources:
-          requests:
-            cpu: "100m"
-            memory: "200M"
-          limits:
-            cpu: "400m"
-            memory: "512M"
-```
-
-You would need to save this file as `POD-TEMPLATE-NAME`. Once you have a valid Pod Template you may proceed to generate a `ConfigMap`.
-
-### Configuration Maps
-
-> You must place the `ConfigMap` in the same namespace as the Heron API Server using the `--namespace` option in the commands below if the server is not in the `default` namespace.
-
-To generate a `ConfigMap` you will need to run the following command:
-
-```bash
-kubectl create configmap CONFIG-MAP-NAME --from-file path/to/POD-TEMPLATE-NAME
-```
-
-You may then want to verify the contents of the `ConfigMap` by running the following command:
-
-```bash
-kubectl get configmaps CONFIG-MAP-NAME -o yaml
-```
-
-The `ConfigMap` should appear similar to the one below for our example:
-
-```yaml
-apiVersion: v1
-data:
-  POD-TEMPLATE-NAME: |
-    apiVersion: v1
-    kind: PodTemplate
-    metadata:
-      name: heron-tracker
-      namespace: default
-    template:
-      metadata:
-        labels:
-          app: heron-tracker
-      spec:
-        containers:
-          - name: heron-tracker
-            image: apache/heron:latest
-            ports:
-              - containerPort: 8888
-                name: api-port
-            resources:
-              requests:
-                cpu: "100m"
-                memory: "200M"
-              limits:
-                cpu: "400m"
-                memory: "512M"
-kind: ConfigMap
-metadata:
-  creationTimestamp: "2021-09-27T21:55:30Z"
-  name: CONFIG-MAP-NAME
-  namespace: default
-  resourceVersion: "1313"
-  uid: ba001653-03d9-4ac8-804c-d2c55c974281
-```
-
-## Submitting
-
-To use the `ConfigMap` for a topology you would submit with the additional flag `--confg-property`. The `--config-property key=value` takes a key value pair:
-
-* Key: `heron.kubernetes.pod.template.configmap.name`
-* Value: `CONFIG-MAP-NAME.POD-TEMPLATE-NAME`
-
-Please note that you must concatenate `CONFIG-MAP-NAME` and `POD-TEMPLATE-NAME` with a **`.`** (period chracter).
-
-For example:
-
-```bash
-heron submit kubernetes \
-  --service-url=http://localhost:8001/api/v1/namespaces/default/services/heron-apiserver:9000/proxy \
-  ~/.heron/examples/heron-api-examples.jar \
-  org.apache.heron.examples.api.AckingTopology acking \
-  --config-property heron.kubernetes.pod.template.configmap.name=CONFIG-MAP-NAME.POD-TEMPLATE-NAME
-```
-
-## Heron Configured Items in Pod Templates
-
-Heron will locate the container named `executor` in the Pod Template and customize it as outlined below. All other containers within the Pod Template will remain unchanged.
-
-### Executor Container
-
-All metadata for the `executor` container will be overwritten by Heron. In some other cases, values from the Pod Template for the `executor` will be overwritten by Heron as outline below.
-
-| Name | Description | Policy |
-|---|---|---|
-| `image` | The `executor` container's image. | Overwritten by Heron using values form the config.
-| `env` | Environment variables are made available within the container. The `HOST` and `POD_NAME` keys are required by Heron and are thus reserved. | Merged with Heron's values taking precedence. Deduplication is based on `name`.
-| `ports` | Port numbers opened within the container. Some of these port number are required by Heron and are thus reserved. The reserved ports are defined in Heron's constants as [`6001`-`6010`]. | Merged with Heron's values taking precedence. Deduplication is based on the `containerPort` value.
-| `limits` | Heron will attempt to load values for `cpu` and `memory` from configs. If these values are not provided in the Configs, then values from the Pod Templates will be used. | Heron's values take precedence over those in the Pod Templates.
-| `volumeMounts` | These are the mount points within the `executor` container for the `volumes` available in the Pod. | Merged with Heron's values taking precedence. Deduplication is based on the `name` value.
-| Annotation: `prometheus.io/scrape` | Flag to indicate whether Prometheus logs can be scraped and is set to `true`. | Value is overridden by Heron. |
-| Annotation `prometheus.io/port` | Port address for Prometheus log scraping and is set to `8080`. | Values are overridden by Heron.
-| Annotation: Pod | Pod's revision/version hash. | Automatically set.
-| Annotation: Service | Labels services can use to attach to the Pod. | Automatically set.
-| Label: `app` | Name of the application lauching the Pod and is set to `Heron`. | Values are overridden by Heron.
-| Label: `topology`| The name of topology which was provided when submitting. | User defined and supplied on the CLI.
-
-### Pod
-
-The following items will be set in the Pod Template's `spec` by Heron.
-
-| Name | Description | Policy |
-|---|---|---|
-`terminationGracePeriodSeconds` | Grace period to wait before shutting down the Pod after a `SIGTERM` signal and is set to `0` seconds. | Values are overridden by Heron.
-| `tolerations` | Attempts to schedule Pods with `taints` onto nodes hosting Pods with matching `taints`. The entries below are included by default. <br>  Keys:<br>`node.kubernetes.io/not-ready` <br> `node.kubernetes.io/unreachable` <br> Values (common):<br> `operator: Exists`<br> `effect: NoExecute`<br> `tolerationSeconds: 10L` | Merged with Heron's values taking precedence. Deduplication is based on the `key` value.
-| `containers` | Configurations for containers to be launched within the Pod. | All `containers`, excluding the `executor`, are loaded as-is.
-| `volumes` | Volumes to be made available to the entire Pod. | Merged with Heron's values taking precedence. Deduplication is based on the `name` value.
-| `secretVolumes` | Secrets to be mounted as volumes within the Pod. | Loaded from the Heron configs if present.
diff --git a/website2/website/sidebars.json b/website2/website/sidebars.json
index c803fdf3a96..50c2086ef5e 100755
--- a/website2/website/sidebars.json
+++ b/website2/website/sidebars.json
@@ -53,8 +53,7 @@
     "Schedulers": [
       "schedulers-k8s-by-hand",
       "schedulers-k8s-with-helm",
-      "schedulers-k8s-pod-templates",
-      "schedulers-k8s-persistent-volume-claims",
+      "schedulers-k8s-execution-environment",
       "schedulers-aurora-cluster",
       "schedulers-aurora-local",
       "schedulers-local",


Test patch:
diff --git a/heron/schedulers/tests/java/org/apache/heron/scheduler/kubernetes/KubernetesContextTest.java b/heron/schedulers/tests/java/org/apache/heron/scheduler/kubernetes/KubernetesContextTest.java
index 95de4a8b425..988456fa1c8 100644
--- a/heron/schedulers/tests/java/org/apache/heron/scheduler/kubernetes/KubernetesContextTest.java
+++ b/heron/schedulers/tests/java/org/apache/heron/scheduler/kubernetes/KubernetesContextTest.java
@@ -27,6 +27,7 @@
 import org.junit.Assert;
 import org.junit.Test;
 
+import org.apache.heron.common.basics.Pair;
 import org.apache.heron.scheduler.TopologySubmissionException;
 import org.apache.heron.scheduler.kubernetes.KubernetesUtils.TestTuple;
 import org.apache.heron.spi.common.Config;
@@ -36,38 +37,48 @@
 public class KubernetesContextTest {
 
   private static final String TOPOLOGY_NAME = "Topology-Name";
-  private static final String KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME =
-      "heron.kubernetes.pod.template.configmap.name";
   private static final String POD_TEMPLATE_CONFIGMAP_NAME = "pod-template-configmap-name";
   private final Config config = Config.newBuilder().build();
   private final Config configWithPodTemplateConfigMap = Config.newBuilder()
-      .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME,
+      .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_LOCATION,
           POD_TEMPLATE_CONFIGMAP_NAME)
       .build();
 
   @Test
   public void testPodTemplateConfigMapName() {
-    Assert.assertEquals(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME,
-        KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME);
-    Assert.assertEquals(
-        KubernetesContext.getPodTemplateConfigMapName(configWithPodTemplateConfigMap),
-        POD_TEMPLATE_CONFIGMAP_NAME);
-    Assert.assertNull(KubernetesContext.getPodTemplateConfigMapName(config));
+    final String executorKey = String.format(KubernetesContext.KUBERNETES_POD_TEMPLATE_LOCATION,
+        KubernetesConstants.EXECUTOR_NAME);
+    final Config configExecutor = Config.newBuilder()
+        .put(executorKey, POD_TEMPLATE_CONFIGMAP_NAME)
+        .build();
+    Assert.assertEquals("Executor location", POD_TEMPLATE_CONFIGMAP_NAME,
+        KubernetesContext.getPodTemplateConfigMapName(configExecutor, true));
+
+    final String managerKey = String.format(KubernetesContext.KUBERNETES_POD_TEMPLATE_LOCATION,
+        KubernetesConstants.MANAGER_NAME);
+    final Config configManager = Config.newBuilder()
+        .put(managerKey, POD_TEMPLATE_CONFIGMAP_NAME)
+        .build();
+    Assert.assertEquals("Manager location", POD_TEMPLATE_CONFIGMAP_NAME,
+        KubernetesContext.getPodTemplateConfigMapName(configManager, false));
+
+    Assert.assertNull("No Pod Template",
+        KubernetesContext.getPodTemplateConfigMapName(config, true));
   }
 
   @Test
-  public void testPodTemplateConfigMapDisabled() {
-    Assert.assertFalse(KubernetesContext.getPodTemplateConfigMapDisabled(config));
+  public void testPodTemplateDisabled() {
+    Assert.assertFalse(KubernetesContext.getPodTemplateDisabled(config));
     Assert.assertFalse(KubernetesContext
-        .getPodTemplateConfigMapDisabled(configWithPodTemplateConfigMap));
+        .getPodTemplateDisabled(configWithPodTemplateConfigMap));
 
     final Config configWithPodTemplateConfigMapOff = Config.newBuilder()
-        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME,
+        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_LOCATION,
             POD_TEMPLATE_CONFIGMAP_NAME)
-        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_DISABLED, "TRUE")
+        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_DISABLED, "TRUE")
         .build();
     Assert.assertTrue(KubernetesContext
-        .getPodTemplateConfigMapDisabled(configWithPodTemplateConfigMapOff));
+        .getPodTemplateDisabled(configWithPodTemplateConfigMapOff));
   }
 
   @Test
@@ -77,7 +88,7 @@ public void testPersistentVolumeClaimDisabled() {
         .getPersistentVolumeClaimDisabled(configWithPodTemplateConfigMap));
 
     final Config configWithPodTemplateConfigMapOff = Config.newBuilder()
-        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME,
+        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_LOCATION,
             POD_TEMPLATE_CONFIGMAP_NAME)
         .put(KubernetesContext.KUBERNETES_PERSISTENT_VOLUME_CLAIMS_CLI_DISABLED, "TRUE")
         .build();
@@ -90,54 +101,88 @@ public void testGetVolumeClaimTemplates() {
     final String volumeNameOne = "volume-name-one";
     final String volumeNameTwo = "volume-name-two";
     final String claimName = "OnDeMaNd";
-    final String keyPattern = KubernetesContext.KUBERNETES_VOLUME_CLAIM_PREFIX + "%s.%s";
+    final String keyPattern = KubernetesContext.KUBERNETES_VOLUME_CLAIM_PREFIX + "%%s.%%s";
+    final String keyExecutor = String.format(keyPattern, KubernetesConstants.EXECUTOR_NAME);
+    final String keyManager = String.format(keyPattern, KubernetesConstants.MANAGER_NAME);
 
     final String storageClassField = VolumeClaimTemplateConfigKeys.storageClassName.name();
     final String pathField = VolumeClaimTemplateConfigKeys.path.name();
     final String claimNameField = VolumeClaimTemplateConfigKeys.claimName.name();
     final String expectedStorageClass = "expected-storage-class";
-    final String storageClassKeyOne = String.format(keyPattern, volumeNameOne, storageClassField);
-    final String storageClassKeyTwo = String.format(keyPattern, volumeNameTwo, storageClassField);
     final String expectedPath = "/path/for/volume/expected";
-    final String pathKeyOne = String.format(keyPattern, volumeNameOne, pathField);
-    final String pathKeyTwo = String.format(keyPattern, volumeNameTwo, pathField);
-    final String claimNameKeyOne = String.format(keyPattern, volumeNameOne, claimNameField);
-    final String claimNameKeyTwo = String.format(keyPattern, volumeNameTwo, claimNameField);
-
-    final Config configPVC = Config.newBuilder()
-        .put(pathKeyOne, expectedPath)
-        .put(pathKeyTwo, expectedPath)
-        .put(claimNameKeyOne, claimName)
-        .put(claimNameKeyTwo, claimName)
-        .put(storageClassKeyOne, expectedStorageClass)
-        .put(storageClassKeyTwo, expectedStorageClass)
-        .build();
 
-    final List<String> expectedKeys = Arrays.asList(volumeNameOne, volumeNameTwo);
-    final List<VolumeClaimTemplateConfigKeys> expectedOptionsKeys =
-        Arrays.asList(VolumeClaimTemplateConfigKeys.path,
-            VolumeClaimTemplateConfigKeys.storageClassName,
-            VolumeClaimTemplateConfigKeys.claimName);
-    final List<String> expectedOptionsValues =
-        Arrays.asList(expectedPath, expectedStorageClass, claimName);
-
-    // List of provided PVC options.
-    final Map<String, Map<VolumeClaimTemplateConfigKeys, String>> mapOfPVC =
-        KubernetesContext.getVolumeClaimTemplates(configPVC);
-
-    Assert.assertTrue("Contains all provided Volumes",
-        mapOfPVC.keySet().containsAll(expectedKeys));
-    for (Map<VolumeClaimTemplateConfigKeys, String> items : mapOfPVC.values()) {
-      Assert.assertTrue("Contains all provided option keys",
-          items.keySet().containsAll(expectedOptionsKeys));
-      Assert.assertTrue("Contains all provided option values",
-          items.values().containsAll(expectedOptionsValues));
+
+    // Test case container.
+    // Input: Config to extract options from, Boolean to indicate Manager/Executor.
+    // Output: [0] expectedKeys, [1] expectedOptionsKeys, [2] expectedOptionsValues.
+    final List<TestTuple<Pair<Config, Boolean>, Object[]>> testCases = new LinkedList<>();
+
+    // Create test cases for Executor/Manager on even/odd indices respectively.
+    for (int idx = 0; idx < 2; ++idx) {
+
+      // Manager case is default.
+      boolean isExecutor = false;
+      String key = keyManager;
+      String description = KubernetesConstants.MANAGER_NAME;
+
+      // Executor case.
+      if (idx % 2 == 0) {
+        isExecutor = true;
+        key = keyExecutor;
+        description = KubernetesConstants.EXECUTOR_NAME;
+      }
+
+      final String storageClassKeyOne = String.format(key, volumeNameOne, storageClassField);
+      final String storageClassKeyTwo = String.format(key, volumeNameTwo, storageClassField);
+      final String pathKeyOne = String.format(key, volumeNameOne, pathField);
+      final String pathKeyTwo = String.format(key, volumeNameTwo, pathField);
+      final String claimNameKeyOne = String.format(key, volumeNameOne, claimNameField);
+      final String claimNameKeyTwo = String.format(key, volumeNameTwo, claimNameField);
+
+      final Config configPVC = Config.newBuilder()
+          .put(pathKeyOne, expectedPath)
+          .put(pathKeyTwo, expectedPath)
+          .put(claimNameKeyOne, claimName)
+          .put(claimNameKeyTwo, claimName)
+          .put(storageClassKeyOne, expectedStorageClass)
+          .put(storageClassKeyTwo, expectedStorageClass)
+          .build();
+
+      final List<String> expectedKeys = Arrays.asList(volumeNameOne, volumeNameTwo);
+      final List<VolumeClaimTemplateConfigKeys> expectedOptionsKeys =
+          Arrays.asList(VolumeClaimTemplateConfigKeys.path,
+              VolumeClaimTemplateConfigKeys.storageClassName,
+              VolumeClaimTemplateConfigKeys.claimName);
+      final List<String> expectedOptionsValues =
+          Arrays.asList(expectedPath, expectedStorageClass, claimName);
+
+      testCases.add(new TestTuple<>(description,
+          new Pair<>(configPVC, isExecutor),
+          new Object[]{expectedKeys, expectedOptionsKeys, expectedOptionsValues}));
+    }
+
+    // Test loop.
+    for (TestTuple<Pair<Config, Boolean>, Object[]> testCase : testCases) {
+      final Map<String, Map<VolumeClaimTemplateConfigKeys, String>> mapOfPVC =
+          KubernetesContext.getVolumeClaimTemplates(testCase.input.first, testCase.input.second);
+
+      Assert.assertTrue(testCase.description + ": Contains all provided Volumes",
+          mapOfPVC.keySet().containsAll((List<String>) testCase.expected[0]));
+      for (Map<VolumeClaimTemplateConfigKeys, String> items : mapOfPVC.values()) {
+        Assert.assertTrue(testCase.description + ": Contains all provided option keys",
+            items.keySet().containsAll((List<VolumeClaimTemplateConfigKeys>) testCase.expected[1]));
+        Assert.assertTrue(testCase.description + ": Contains all provided option values",
+            items.values().containsAll((List<String>) testCase.expected[2]));
+      }
     }
 
     // Empty PVC.
-    final Map<String, Map<VolumeClaimTemplateConfigKeys, String>> emptyPVC =
-        KubernetesContext.getVolumeClaimTemplates(Config.newBuilder().build());
-    Assert.assertTrue("Empty PVC is returned when no options provided", emptyPVC.isEmpty());
+    final Boolean[] emptyPVCTestCases = new Boolean[] {true, false};
+    for (boolean testCase : emptyPVCTestCases) {
+      final Map<String, Map<VolumeClaimTemplateConfigKeys, String>> emptyPVC =
+          KubernetesContext.getVolumeClaimTemplates(Config.newBuilder().build(), testCase);
+      Assert.assertTrue("Empty PVC is returned when no options provided", emptyPVC.isEmpty());
+    }
   }
 
   @Test
@@ -146,8 +191,8 @@ public void testGetPersistentVolumeClaimsErrors() {
     final String volumeNameInvalid = "volume-Name-Invalid";
     final String failureValue = "Should-Fail";
     final String generalFailureMessage = "Invalid Persistent Volume";
-    final String keyPattern = KubernetesContext.KUBERNETES_VOLUME_CLAIM_PREFIX
-        + "%s.%s";
+    final String keyPattern = String.format(KubernetesContext.KUBERNETES_VOLUME_CLAIM_PREFIX
+        + "%%s.%%s", KubernetesConstants.EXECUTOR_NAME);
     final List<TestTuple<Config, String>> testCases = new LinkedList<>();
 
     // Invalid option key test.
@@ -186,11 +231,15 @@ public void testGetPersistentVolumeClaimsErrors() {
         configInvalidStorageClassName, "Option `storageClassName`"));
 
     // Testing loop.
+    final Boolean[] executorFlags = new Boolean[] {true, false};
     for (TestTuple<Config, String> testCase : testCases) {
-      try {
-        KubernetesContext.getVolumeClaimTemplates(testCase.input);
-      } catch (TopologySubmissionException e) {
-        Assert.assertTrue(testCase.description, e.getMessage().contains(testCase.expected));
+      // Test for both Executor and Manager.
+      for (boolean isExecutor : executorFlags) {
+        try {
+          KubernetesContext.getVolumeClaimTemplates(testCase.input, isExecutor);
+        } catch (TopologySubmissionException e) {
+          Assert.assertTrue(testCase.description, e.getMessage().contains(testCase.expected));
+        }
       }
     }
   }
diff --git a/heron/schedulers/tests/java/org/apache/heron/scheduler/kubernetes/V1ControllerTest.java b/heron/schedulers/tests/java/org/apache/heron/scheduler/kubernetes/V1ControllerTest.java
index b543ea905a5..e79760d6600 100644
--- a/heron/schedulers/tests/java/org/apache/heron/scheduler/kubernetes/V1ControllerTest.java
+++ b/heron/schedulers/tests/java/org/apache/heron/scheduler/kubernetes/V1ControllerTest.java
@@ -103,132 +103,167 @@ public class V1ControllerTest {
           + "          limits:\n"
           + "            cpu: \"400m\"\n"
           + "            memory: \"512M\"";
-
-  private final Config config = Config.newBuilder().build();
-  private final Config configWithPodTemplate = Config.newBuilder()
-      .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME, CONFIGMAP_POD_TEMPLATE_NAME)
+  private static final String POD_TEMPLATE_LOCATION_EXECUTOR =
+      String.format(KubernetesContext.KUBERNETES_POD_TEMPLATE_LOCATION,
+          KubernetesConstants.EXECUTOR_NAME);
+  private static final String POD_TEMPLATE_LOCATION_MANAGER =
+      String.format(KubernetesContext.KUBERNETES_POD_TEMPLATE_LOCATION,
+          KubernetesConstants.MANAGER_NAME);
+
+  private static final Config CONFIG = Config.newBuilder().build();
+  private static final Config CONFIG_WITH_POD_TEMPLATE = Config.newBuilder()
+      .put(POD_TEMPLATE_LOCATION_EXECUTOR, CONFIGMAP_POD_TEMPLATE_NAME)
+      .put(POD_TEMPLATE_LOCATION_MANAGER, CONFIGMAP_POD_TEMPLATE_NAME)
       .build();
-  private final Config runtime = Config.newBuilder()
+  private static final Config RUNTIME = Config.newBuilder()
       .put(Key.TOPOLOGY_NAME, TOPOLOGY_NAME)
       .build();
   private final Config configDisabledPodTemplate = Config.newBuilder()
-      .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME, CONFIGMAP_POD_TEMPLATE_NAME)
-      .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_DISABLED, "true")
+      .put(POD_TEMPLATE_LOCATION_EXECUTOR, CONFIGMAP_POD_TEMPLATE_NAME)
+      .put(POD_TEMPLATE_LOCATION_MANAGER, CONFIGMAP_POD_TEMPLATE_NAME)
+      .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_DISABLED, "true")
       .build();
 
   @Spy
   private final V1Controller v1ControllerWithPodTemplate =
-      new V1Controller(configWithPodTemplate, runtime);
+      new V1Controller(CONFIG_WITH_POD_TEMPLATE, RUNTIME);
 
   @Spy
   private final V1Controller v1ControllerPodTemplate =
-      new V1Controller(configDisabledPodTemplate, runtime);
+      new V1Controller(configDisabledPodTemplate, RUNTIME);
 
   @Rule
   public final ExpectedException expectedException = ExpectedException.none();
 
   @Test
   public void testLoadPodFromTemplateDefault() {
-    final V1Controller v1ControllerNoPodTemplate = new V1Controller(config, runtime);
-    final V1PodTemplateSpec podSpec = v1ControllerNoPodTemplate.loadPodFromTemplate();
+    final V1Controller v1ControllerNoPodTemplate = new V1Controller(CONFIG, RUNTIME);
+    final V1PodTemplateSpec defaultPodSpec = new V1PodTemplateSpec();
+
+    final V1PodTemplateSpec podSpecExecutor = v1ControllerNoPodTemplate.loadPodFromTemplate(true);
+    Assert.assertEquals("Default Pod Spec for Executor", defaultPodSpec, podSpecExecutor);
 
-    Assert.assertEquals(podSpec, new V1PodTemplateSpec());
+    final V1PodTemplateSpec podSpecManager = v1ControllerNoPodTemplate.loadPodFromTemplate(false);
+    Assert.assertEquals("Default Pod Spec for Manager", defaultPodSpec, podSpecManager);
   }
 
   @Test
   public void testLoadPodFromTemplateNullConfigMap() {
-    final String expected = "unable to locate";
-    String message = "";
+    final List<TestTuple<Boolean, String>> testCases = new LinkedList<>();
+    testCases.add(new TestTuple<>("Executor not found", true, "unable to locate"));
+    testCases.add(new TestTuple<>("Manager not found", false, "unable to locate"));
 
-    doReturn(null)
-        .when(v1ControllerWithPodTemplate)
-        .getConfigMap(anyString());
-    try {
-      v1ControllerWithPodTemplate.loadPodFromTemplate();
-    } catch (TopologySubmissionException e) {
-      message = e.getMessage();
+    for (TestTuple<Boolean, String> testCase : testCases) {
+      doReturn(null)
+          .when(v1ControllerWithPodTemplate)
+          .getConfigMap(anyString());
+
+      String message = "";
+      try {
+        v1ControllerWithPodTemplate.loadPodFromTemplate(testCase.input);
+      } catch (TopologySubmissionException e) {
+        message = e.getMessage();
+      }
+      Assert.assertTrue(testCase.description, message.contains(testCase.expected));
     }
-    Assert.assertTrue(message.contains(expected));
   }
 
   @Test
   public void testLoadPodFromTemplateNoConfigMap() {
-    final String expected = "Failed to locate Pod Template";
-    String message = "";
+    final List<TestTuple<Boolean, String>> testCases = new LinkedList<>();
+    testCases.add(new TestTuple<>("Executor no ConfigMap", true, "Failed to locate Pod Template"));
+    testCases.add(new TestTuple<>("Manager no ConfigMap", false, "Failed to locate Pod Template"));
 
-    doReturn(new V1ConfigMap())
-        .when(v1ControllerWithPodTemplate)
-        .getConfigMap(anyString());
-    try {
-      v1ControllerWithPodTemplate.loadPodFromTemplate();
-    } catch (TopologySubmissionException e) {
-      message = e.getMessage();
+    for (TestTuple<Boolean, String> testCase : testCases) {
+      doReturn(new V1ConfigMap())
+          .when(v1ControllerWithPodTemplate)
+          .getConfigMap(anyString());
+
+      String message = "";
+      try {
+        v1ControllerWithPodTemplate.loadPodFromTemplate(testCase.input);
+      } catch (TopologySubmissionException e) {
+        message = e.getMessage();
+      }
+      Assert.assertTrue(testCase.description, message.contains(testCase.expected));
     }
-    Assert.assertTrue(message.contains(expected));
   }
 
   @Test
   public void testLoadPodFromTemplateNoTargetConfigMap() {
-    final String expected = "Failed to locate Pod Template";
-    String message = "";
-    V1ConfigMap configMapNoTargetData = new V1ConfigMapBuilder()
+    final List<TestTuple<Boolean, String>> testCases = new LinkedList<>();
+    testCases.add(new TestTuple<>("Executor no target ConfigMap",
+        true, "Failed to locate Pod Template"));
+    testCases.add(new TestTuple<>("Manager no target ConfigMap",
+        false, "Failed to locate Pod Template"));
+
+    final V1ConfigMap configMapNoTargetData = new V1ConfigMapBuilder()
         .withNewMetadata()
           .withName(CONFIGMAP_NAME)
         .endMetadata()
         .addToData("Dummy Key", "Dummy Value")
         .build();
 
-    doReturn(configMapNoTargetData)
-        .when(v1ControllerWithPodTemplate)
-        .getConfigMap(anyString());
-    try {
-      v1ControllerWithPodTemplate.loadPodFromTemplate();
-    } catch (TopologySubmissionException e) {
-      message = e.getMessage();
+    for (TestTuple<Boolean, String> testCase : testCases) {
+      doReturn(configMapNoTargetData)
+          .when(v1ControllerWithPodTemplate)
+          .getConfigMap(anyString());
+
+      String message = "";
+      try {
+        v1ControllerWithPodTemplate.loadPodFromTemplate(testCase.input);
+      } catch (TopologySubmissionException e) {
+        message = e.getMessage();
+      }
+      Assert.assertTrue(testCase.description, message.contains(testCase.expected));
     }
-    Assert.assertTrue(message.contains(expected));
   }
 
   @Test
   public void testLoadPodFromTemplateBadTargetConfigMap() {
-    final String expected = "Error parsing";
-    String message = "";
-
     // ConfigMap with target ConfigMap and an invalid Pod Template.
-    V1ConfigMap configMapInvalidPod = new V1ConfigMapBuilder()
+    final V1ConfigMap configMapInvalidPod = new V1ConfigMapBuilder()
         .withNewMetadata()
           .withName(CONFIGMAP_NAME)
         .endMetadata()
         .addToData(POD_TEMPLATE_NAME, "Dummy Value")
         .build();
 
-    doReturn(configMapInvalidPod)
-        .when(v1ControllerWithPodTemplate)
-        .getConfigMap(anyString());
-    try {
-      v1ControllerWithPodTemplate.loadPodFromTemplate();
-    } catch (TopologySubmissionException e) {
-      message = e.getMessage();
-    }
-    Assert.assertTrue("Invalid Pod Template parsing should fail", message.contains(expected));
-
     // ConfigMap with target ConfigMaps and an empty Pod Template.
-    V1ConfigMap configMapEmptyPod = new V1ConfigMapBuilder()
+    final V1ConfigMap configMapEmptyPod = new V1ConfigMapBuilder()
         .withNewMetadata()
-          .withName(CONFIGMAP_NAME)
+        .withName(CONFIGMAP_NAME)
         .endMetadata()
         .addToData(POD_TEMPLATE_NAME, "")
         .build();
 
-    doReturn(configMapEmptyPod)
-        .when(v1ControllerWithPodTemplate)
-        .getConfigMap(anyString());
-    try {
-      v1ControllerWithPodTemplate.loadPodFromTemplate();
-    } catch (TopologySubmissionException e) {
-      message = e.getMessage();
+    // Test case container.
+    // Input: ConfigMap to setup mock V1Controller, Boolean flag for executor/manager switch.
+    // Output: The expected error message.
+    final List<TestTuple<Pair<V1ConfigMap, Boolean>, String>> testCases = new LinkedList<>();
+    testCases.add(new TestTuple<>("Executor invalid Pod Template",
+        new Pair<>(configMapInvalidPod, true), "Error parsing"));
+    testCases.add(new TestTuple<>("Manager invalid Pod Template",
+        new Pair<>(configMapInvalidPod, false), "Error parsing"));
+    testCases.add(new TestTuple<>("Executor empty Pod Template",
+        new Pair<>(configMapEmptyPod, true), "Error parsing"));
+    testCases.add(new TestTuple<>("Manager empty Pod Template",
+        new Pair<>(configMapEmptyPod, false), "Error parsing"));
+
+    // Test loop.
+    for (TestTuple<Pair<V1ConfigMap, Boolean>, String> testCase : testCases) {
+      doReturn(testCase.input.first)
+          .when(v1ControllerWithPodTemplate)
+          .getConfigMap(anyString());
+
+      String message = "";
+      try {
+        v1ControllerWithPodTemplate.loadPodFromTemplate(testCase.input.second);
+      } catch (TopologySubmissionException e) {
+        message = e.getMessage();
+      }
+      Assert.assertTrue(testCase.description, message.contains(testCase.expected));
     }
-    Assert.assertTrue("Empty Pod Template parsing should fail", message.contains(expected));
   }
 
   @Test
@@ -272,18 +307,32 @@ public void testLoadPodFromTemplateValidConfigMap() {
 
 
     // ConfigMap with valid Pod Template.
-    V1ConfigMap configMapValidPod = new V1ConfigMapBuilder()
+    final V1ConfigMap configMapValidPod = new V1ConfigMapBuilder()
         .withNewMetadata()
           .withName(CONFIGMAP_NAME)
         .endMetadata()
         .addToData(POD_TEMPLATE_NAME, POD_TEMPLATE_VALID)
         .build();
-    doReturn(configMapValidPod)
-        .when(v1ControllerWithPodTemplate)
-        .getConfigMap(anyString());
-    V1PodTemplateSpec podTemplateSpec = v1ControllerWithPodTemplate.loadPodFromTemplate();
 
-    Assert.assertTrue(podTemplateSpec.toString().contains(expected));
+    // Test case container.
+    // Input: ConfigMap to setup mock V1Controller, Boolean flag for executor/manager switch.
+    // Output: The expected Pod template as a string.
+    final List<TestTuple<Pair<V1ConfigMap, Boolean>, String>> testCases = new LinkedList<>();
+    testCases.add(new TestTuple<>("Executor valid Pod Template",
+        new Pair<>(configMapValidPod, true), expected));
+    testCases.add(new TestTuple<>("Manager valid Pod Template",
+        new Pair<>(configMapValidPod, false), expected));
+
+    // Test loop.
+    for (TestTuple<Pair<V1ConfigMap, Boolean>, String> testCase : testCases) {
+      doReturn(testCase.input.first)
+          .when(v1ControllerWithPodTemplate)
+          .getConfigMap(anyString());
+
+      V1PodTemplateSpec podTemplateSpec = v1ControllerWithPodTemplate.loadPodFromTemplate(true);
+
+      Assert.assertTrue(podTemplateSpec.toString().contains(testCase.expected));
+    }
   }
 
   @Test
@@ -300,24 +349,37 @@ public void testLoadPodFromTemplateInvalidConfigMap() {
             + "    labels:\n"
             + "      app: heron-tracker\n"
             + "  spec:\n";
-    V1ConfigMap configMap = new V1ConfigMapBuilder()
+    final V1ConfigMap configMap = new V1ConfigMapBuilder()
         .withNewMetadata()
           .withName(CONFIGMAP_NAME)
         .endMetadata()
         .addToData(POD_TEMPLATE_NAME, invalidPodTemplate)
         .build();
-    final String expected = "Error parsing";
-    String message = "";
 
-    doReturn(configMap)
-        .when(v1ControllerWithPodTemplate)
-        .getConfigMap(anyString());
-    try {
-      v1ControllerWithPodTemplate.loadPodFromTemplate();
-    } catch (TopologySubmissionException e) {
-      message = e.getMessage();
+
+    // Test case container.
+    // Input: ConfigMap to setup mock V1Controller, Boolean flag for executor/manager switch.
+    // Output: The expected Pod template as a string.
+    final List<TestTuple<Pair<V1ConfigMap, Boolean>, String>> testCases = new LinkedList<>();
+    testCases.add(new TestTuple<>("Executor invalid Pod Template",
+        new Pair<>(configMap, true), "Error parsing"));
+    testCases.add(new TestTuple<>("Manager invalid Pod Template",
+        new Pair<>(configMap, false), "Error parsing"));
+
+    // Test loop.
+    for (TestTuple<Pair<V1ConfigMap, Boolean>, String> testCase : testCases) {
+      doReturn(testCase.input.first)
+          .when(v1ControllerWithPodTemplate)
+          .getConfigMap(anyString());
+
+      String message = "";
+      try {
+        v1ControllerWithPodTemplate.loadPodFromTemplate(testCase.input.second);
+      } catch (TopologySubmissionException e) {
+        message = e.getMessage();
+      }
+      Assert.assertTrue(message.contains(testCase.expected));
     }
-    Assert.assertTrue(message.contains(expected));
   }
 
   @Test
@@ -336,7 +398,7 @@ public void testDisablePodTemplates() {
         .getConfigMap(anyString());
 
     try {
-      v1ControllerPodTemplate.loadPodFromTemplate();
+      v1ControllerPodTemplate.loadPodFromTemplate(true);
     } catch (TopologySubmissionException e) {
       message = e.getMessage();
     }
@@ -346,45 +408,41 @@ public void testDisablePodTemplates() {
   @Test
   public void testGetPodTemplateLocationPassing() {
     final Config testConfig = Config.newBuilder()
-        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME, CONFIGMAP_POD_TEMPLATE_NAME)
+        .put(POD_TEMPLATE_LOCATION_EXECUTOR, CONFIGMAP_POD_TEMPLATE_NAME)
         .build();
-    final V1Controller v1Controller = new V1Controller(testConfig, runtime);
+    final V1Controller v1Controller = new V1Controller(testConfig, RUNTIME);
     final Pair<String, String> expected = new Pair<>(CONFIGMAP_NAME, POD_TEMPLATE_NAME);
-    Pair<String, String> actual;
 
     // Correct parsing
-    actual = v1Controller.getPodTemplateLocation();
-    Assert.assertEquals(actual, expected);
+    final Pair<String, String> actual = v1Controller.getPodTemplateLocation(true);
+    Assert.assertEquals(expected, actual);
   }
 
   @Test
   public void testGetPodTemplateLocationNoConfigMap() {
     expectedException.expect(TopologySubmissionException.class);
     final Config testConfig = Config.newBuilder()
-        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME,
-        ".POD-TEMPLATE-NAME").build();
-    V1Controller v1Controller = new V1Controller(testConfig, runtime);
-    v1Controller.getPodTemplateLocation();
+        .put(POD_TEMPLATE_LOCATION_EXECUTOR, ".POD-TEMPLATE-NAME").build();
+    V1Controller v1Controller = new V1Controller(testConfig, RUNTIME);
+    v1Controller.getPodTemplateLocation(true);
   }
 
   @Test
   public void testGetPodTemplateLocationNoPodTemplate() {
     expectedException.expect(TopologySubmissionException.class);
     final Config testConfig = Config.newBuilder()
-        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME,
-        "CONFIGMAP-NAME.").build();
-    V1Controller v1Controller = new V1Controller(testConfig, runtime);
-    v1Controller.getPodTemplateLocation();
+        .put(POD_TEMPLATE_LOCATION_EXECUTOR, "CONFIGMAP-NAME.").build();
+    V1Controller v1Controller = new V1Controller(testConfig, RUNTIME);
+    v1Controller.getPodTemplateLocation(true);
   }
 
   @Test
   public void testGetPodTemplateLocationNoDelimiter() {
     expectedException.expect(TopologySubmissionException.class);
     final Config testConfig = Config.newBuilder()
-        .put(KubernetesContext.KUBERNETES_POD_TEMPLATE_CONFIGMAP_NAME,
-        "CONFIGMAP-NAMEPOD-TEMPLATE-NAME").build();
-    V1Controller v1Controller = new V1Controller(testConfig, runtime);
-    v1Controller.getPodTemplateLocation();
+        .put(POD_TEMPLATE_LOCATION_EXECUTOR, "CONFIGMAP-NAMEPOD-TEMPLATE-NAME").build();
+    V1Controller v1Controller = new V1Controller(testConfig, RUNTIME);
+    v1Controller.getPodTemplateLocation(true);
   }
 
   @Test
@@ -499,10 +557,12 @@ public void testConfigureContainerEnvVars() {
 
   @Test
   public void testConfigureContainerResources() {
+    final boolean isExecutor = true;
+
     final Resource resourceDefault = new Resource(
-        9, ByteAmount.fromGigabytes(19), ByteAmount.fromGigabytes(99));
+        9, ByteAmount.fromMegabytes(19000), ByteAmount.fromMegabytes(99000));
     final Resource resourceCustom = new Resource(
-        4, ByteAmount.fromGigabytes(34), ByteAmount.fromGigabytes(400));
+        4, ByteAmount.fromMegabytes(34000), ByteAmount.fromMegabytes(400000));
 
     final Quantity defaultRAM = Quantity.fromString(
         KubernetesUtils.Megabytes(resourceDefault.getRam()));
@@ -513,7 +573,7 @@ public void testConfigureContainerResources() {
     final Quantity customCPU = Quantity.fromString(
         Double.toString(V1Controller.roundDecimal(resourceCustom.getCpu(), 3)));
     final Quantity customDisk = Quantity.fromString(
-        Double.toString(V1Controller.roundDecimal(resourceCustom.getDisk().getValue(), 3)));
+        KubernetesUtils.Megabytes(resourceCustom.getDisk()));
 
     final Config configNoLimit = Config.newBuilder()
         .put(KubernetesContext.KUBERNETES_RESOURCE_REQUEST_MODE, "NOT_SET")
@@ -539,7 +599,7 @@ public void testConfigureContainerResources() {
     // Default. Null resources.
     V1Container containerNull = new V1ContainerBuilder().build();
     v1ControllerWithPodTemplate.configureContainerResources(
-        containerNull, configNoLimit, resourceDefault);
+        containerNull, configNoLimit, resourceDefault, isExecutor);
     Assert.assertTrue("Default LIMITS should be set in container with null LIMITS",
         containerNull.getResources().getLimits().entrySet()
             .containsAll(expectDefaultRequirements.getLimits().entrySet()));
@@ -547,7 +607,7 @@ public void testConfigureContainerResources() {
     // Empty resources.
     V1Container containerEmpty = new V1ContainerBuilder().withNewResources().endResources().build();
     v1ControllerWithPodTemplate.configureContainerResources(
-        containerEmpty, configNoLimit, resourceDefault);
+        containerEmpty, configNoLimit, resourceDefault, isExecutor);
     Assert.assertTrue("Default LIMITS should be set in container with empty LIMITS",
         containerNull.getResources().getLimits().entrySet()
             .containsAll(expectDefaultRequirements.getLimits().entrySet()));
@@ -557,7 +617,7 @@ public void testConfigureContainerResources() {
         .withResources(customRequirements)
         .build();
     v1ControllerWithPodTemplate.configureContainerResources(
-        containerCustom, configNoLimit, resourceDefault);
+        containerCustom, configNoLimit, resourceDefault, isExecutor);
     Assert.assertTrue("Custom LIMITS should be set in container with custom LIMITS",
         containerCustom.getResources().getLimits().entrySet()
             .containsAll(expectCustomRequirements.getLimits().entrySet()));
@@ -567,7 +627,7 @@ public void testConfigureContainerResources() {
         .withResources(customRequirements)
         .build();
     v1ControllerWithPodTemplate.configureContainerResources(
-        containerRequests, configWithLimit, resourceDefault);
+        containerRequests, configWithLimit, resourceDefault, isExecutor);
     Assert.assertTrue("Custom LIMITS should be set in container with custom LIMITS and REQUEST",
         containerRequests.getResources().getLimits().entrySet()
             .containsAll(expectCustomRequirements.getLimits().entrySet()));
@@ -576,6 +636,49 @@ public void testConfigureContainerResources() {
             .containsAll(expectCustomRequirements.getLimits().entrySet()));
   }
 
+  @Test
+  public void testConfigureContainerResourcesCLI() {
+    final boolean isExecutor = true;
+    final String customLimitMEMStr = "120Gi";
+    final String customLimitCPUStr = "5";
+    final String customRequestMEMStr = "100Mi";
+    final String customRequestCPUStr = "4";
+
+    final Resource resources = new Resource(
+        6, ByteAmount.fromMegabytes(34000), ByteAmount.fromGigabytes(400));
+
+    final Quantity customLimitMEM = Quantity.fromString(customLimitMEMStr);
+    final Quantity customLimitCPU = Quantity.fromString(customLimitCPUStr);
+    final Quantity customRequestMEM = Quantity.fromString(customRequestMEMStr);
+    final Quantity customRequestCPU = Quantity.fromString(customRequestCPUStr);
+
+    final Config config = Config.newBuilder()
+        .put(String.format(KubernetesContext.KUBERNETES_RESOURCE_LIMITS_PREFIX
+                + KubernetesConstants.CPU, KubernetesConstants.EXECUTOR_NAME), customLimitCPUStr)
+        .put(String.format(KubernetesContext.KUBERNETES_RESOURCE_LIMITS_PREFIX
+                + KubernetesConstants.MEMORY, KubernetesConstants.EXECUTOR_NAME), customLimitMEMStr)
+        .put(String.format(KubernetesContext.KUBERNETES_RESOURCE_REQUESTS_PREFIX
+                + KubernetesConstants.CPU, KubernetesConstants.EXECUTOR_NAME), customRequestCPUStr)
+        .put(String.format(KubernetesContext.KUBERNETES_RESOURCE_REQUESTS_PREFIX
+                + KubernetesConstants.MEMORY, KubernetesConstants.EXECUTOR_NAME),
+            customRequestMEMStr)
+        .put(KubernetesContext.KUBERNETES_RESOURCE_REQUEST_MODE, "EQUAL_TO_LIMIT")
+        .build();
+
+    final V1Container expected = new V1ContainerBuilder()
+        .withNewResources()
+          .addToLimits(KubernetesConstants.CPU, customLimitCPU)
+          .addToLimits(KubernetesConstants.MEMORY, customLimitMEM)
+          .addToRequests(KubernetesConstants.CPU, customRequestCPU)
+          .addToRequests(KubernetesConstants.MEMORY, customRequestMEM)
+        .endResources()
+        .build();
+
+    final V1Container actual = new V1Container();
+    v1ControllerWithPodTemplate.configureContainerResources(actual, config, resources, isExecutor);
+    Assert.assertEquals("Container Resources are set from CLI.", expected, actual);
+  }
+
   @Test
   public void testAddVolumesIfPresent() {
     final String pathDefault = "config-host-volume-path";
@@ -585,7 +688,7 @@ public void testAddVolumesIfPresent() {
         .put(KubernetesContext.KUBERNETES_VOLUME_TYPE, Volumes.HOST_PATH)
         .put(KubernetesContext.KUBERNETES_VOLUME_HOSTPATH_PATH, pathDefault)
         .build();
-    final V1Controller controllerWithVol = new V1Controller(configWithVolumes, runtime);
+    final V1Controller controllerWithVol = new V1Controller(configWithVolumes, RUNTIME);
 
     final V1Volume volumeDefault = new V1VolumeBuilder()
         .withName(pathNameDefault)
@@ -613,7 +716,7 @@ public void testAddVolumesIfPresent() {
     final List<V1Volume> expectedCustom = Arrays.asList(volumeDefault, volumeToBeKept);
 
     // No Volumes set.
-    V1Controller controllerDoNotSetVolumes = new V1Controller(Config.newBuilder().build(), runtime);
+    V1Controller controllerDoNotSetVolumes = new V1Controller(Config.newBuilder().build(), RUNTIME);
     V1PodSpec podSpecNoSetVolumes = new V1PodSpec();
     controllerDoNotSetVolumes.addVolumesIfPresent(podSpecNoSetVolumes);
     Assert.assertNull(podSpecNoSetVolumes.getVolumes());
@@ -649,7 +752,7 @@ public void testMountVolumeIfPresent() {
         .put(KubernetesContext.KUBERNETES_CONTAINER_VOLUME_MOUNT_NAME, pathNameDefault)
         .put(KubernetesContext.KUBERNETES_CONTAINER_VOLUME_MOUNT_PATH, pathDefault)
         .build();
-    final V1Controller controllerWithMounts = new V1Controller(configWithVolumes, runtime);
+    final V1Controller controllerWithMounts = new V1Controller(configWithVolumes, RUNTIME);
     final V1VolumeMount volumeDefault = new V1VolumeMountBuilder()
         .withName(pathNameDefault)
         .withMountPath(pathDefault)
@@ -670,7 +773,7 @@ public void testMountVolumeIfPresent() {
     );
 
     // No Volume Mounts set.
-    V1Controller controllerDoNotSetMounts = new V1Controller(Config.newBuilder().build(), runtime);
+    V1Controller controllerDoNotSetMounts = new V1Controller(Config.newBuilder().build(), RUNTIME);
     V1Container containerNoSetMounts = new V1Container();
     controllerDoNotSetMounts.mountVolumeIfPresent(containerNoSetMounts);
     Assert.assertNull(containerNoSetMounts.getVolumeMounts());
@@ -1016,9 +1119,10 @@ public void testConfigurePodWithPersistentVolumeClaims() {
           .when(v1ControllerWithPodTemplate)
           .createPersistentVolumeClaimVolumesAndMounts(anyMap());
 
+      // <configPVC> parameter is used in mock above, so we can set it to <null> as it is not used.
       v1ControllerWithPodTemplate
           .configurePodWithPersistentVolumeClaimVolumesAndMounts((V1PodSpec) testCase.input[0],
-              (V1Container) testCase.input[1]);
+              (V1Container) testCase.input[1], null);
 
       Assert.assertEquals("Pod Specs match " + testCase.description,
           testCase.input[0], testCase.expected.first);
@@ -1026,4 +1130,96 @@ public void testConfigurePodWithPersistentVolumeClaims() {
           testCase.input[1], testCase.expected.second);
     }
   }
+
+  @Test
+  public void testSetShardIdEnvironmentVariableCommand() {
+
+    List<TestTuple<Boolean, String>> testCases = new LinkedList<>();
+
+    testCases.add(new TestTuple<>("Executor command is set correctly",
+        true, "SHARD_ID=$((${POD_NAME##*-} + 1)) && echo shardId=${SHARD_ID}"));
+    testCases.add(new TestTuple<>("Manager command is set correctly",
+        false, "SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID}"));
+
+    for (TestTuple<Boolean, String> testCase : testCases) {
+      Assert.assertEquals(testCase.description, testCase.expected,
+          v1ControllerWithPodTemplate.setShardIdEnvironmentVariableCommand(testCase.input));
+    }
+  }
+
+  @Test
+  public void testCreateResourcesRequirement() {
+    final String managerCpuLimit = "3000m";
+    final String managerMemLimit = "256Gi";
+    final Quantity memory = Quantity.fromString(managerMemLimit);
+    final Quantity cpu = Quantity.fromString(managerCpuLimit);
+    final List<TestTuple<Map<String, String>, Map<String, Quantity>>> testCases =
+        new LinkedList<>();
+
+    // No input.
+    Map<String, String> inputEmpty = new HashMap<>();
+    testCases.add(new TestTuple<>("Empty input.", inputEmpty, new HashMap<>()));
+
+    // Only memory.
+    Map<String, String> inputMemory = new HashMap<String, String>() {
+      {
+        put(KubernetesConstants.MEMORY, managerMemLimit);
+      }
+    };
+    Map<String, Quantity> expectedMemory = new HashMap<String, Quantity>() {
+      {
+        put(KubernetesConstants.MEMORY, memory);
+      }
+    };
+    testCases.add(new TestTuple<>("Only memory input.", inputMemory, expectedMemory));
+
+    // Only CPU.
+    Map<String, String> inputCPU = new HashMap<String, String>() {
+      {
+        put(KubernetesConstants.CPU, managerCpuLimit);
+      }
+    };
+    Map<String, Quantity> expectedCPU = new HashMap<String, Quantity>() {
+      {
+        put(KubernetesConstants.CPU, cpu);
+      }
+    };
+    testCases.add(new TestTuple<>("Only CPU input.", inputCPU, expectedCPU));
+
+    // CPU and memory.
+    Map<String, String> inputMemoryCPU = new HashMap<String, String>() {
+      {
+        put(KubernetesConstants.MEMORY, managerMemLimit);
+        put(KubernetesConstants.CPU, managerCpuLimit);
+      }
+    };
+    Map<String, Quantity> expectedMemoryCPU = new HashMap<String, Quantity>() {
+      {
+        put(KubernetesConstants.MEMORY, memory);
+        put(KubernetesConstants.CPU, cpu);
+      }
+    };
+    testCases.add(new TestTuple<>("Memory and CPU input.", inputMemoryCPU, expectedMemoryCPU));
+
+    // Invalid.
+    Map<String, String> inputInvalid = new HashMap<String, String>() {
+      {
+        put("invalid input", "will not be ignored");
+        put(KubernetesConstants.CPU, managerCpuLimit);
+      }
+    };
+    Map<String, Quantity> expectedInvalid = new HashMap<String, Quantity>() {
+      {
+        put(KubernetesConstants.CPU, cpu);
+      }
+    };
+    testCases.add(new TestTuple<>("Invalid input.", inputInvalid, expectedInvalid));
+
+    // Test loop.
+    for (TestTuple<Map<String, String>, Map<String, Quantity>> testCase : testCases) {
+      Map<String, Quantity> actual =
+          v1ControllerPodTemplate.createResourcesRequirement(testCase.input);
+      Assert.assertEquals(testCase.description, testCase.expected, actual);
+    }
+  }
 }