Repository: keikoproj/upgrade-manager
Pull request #216 (2021-04-16 23:49:36)

Issue 2108: step duration metrics to v1
Related to issue https://github.intuit.com/kubernetes/arktika/issues/2108

Add step duration metrics to v1(followed similar change from v2)
Added service to expose metrics for prometheus

codecov[bot]: # [Codecov](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj) Report
> Merging [#216](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj) (11edf05) into [master](https://codecov.io/gh/keikoproj/upgrade-manager/commit/a1af90bc73ab2bac00f779d71a30c2a0ec12f46d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj) (a1af90b) will **decrease** coverage by `1.01%`.
> The diff coverage is `62.83%`.

[![Impacted file tree graph](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216/graphs/tree.svg?width=650&height=150&src=pr&token=PbUYkNX6wz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj)](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj)

```diff
@@            Coverage Diff             @@
##           master     #216      +/-   ##
==========================================
- Coverage   68.08%   67.06%   -1.02%     
==========================================
  Files          11       12       +1     
  Lines         943     1081     +138     
==========================================
+ Hits          642      725      +83     
- Misses        264      318      +54     
- Partials       37       38       +1     
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `67.06% <62.83%> (-1.02%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj) | Coverage Δ | |
|---|---|---|
| [api/v1alpha1/zz\_generated.deepcopy.go](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj#diff-YXBpL3YxYWxwaGExL3p6X2dlbmVyYXRlZC5kZWVwY29weS5nbw==) | `0.00% <0.00%> (ø)` | |
| [api/v1alpha1/rollingupgrade\_types.go](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj#diff-YXBpL3YxYWxwaGExL3JvbGxpbmd1cGdyYWRlX3R5cGVzLmdv) | `71.18% <71.42%> (+4.51%)` | :arrow_up: |
| [controllers/common/metrics.go](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj#diff-Y29udHJvbGxlcnMvY29tbW9uL21ldHJpY3MuZ28=) | `89.65% <89.65%> (ø)` | |
| [controllers/rollingupgrade\_controller.go](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj#diff-Y29udHJvbGxlcnMvcm9sbGluZ3VwZ3JhZGVfY29udHJvbGxlci5nbw==) | `68.18% <100.00%> (+0.90%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj). Last update [a1af90b...11edf05](https://codecov.io/gh/keikoproj/upgrade-manager/pull/216?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj).


Patch:
diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go
index 42a8a6a..92e55d7 100644
--- a/api/v1alpha1/rollingupgrade_types.go
+++ b/api/v1alpha1/rollingupgrade_types.go
@@ -17,7 +17,9 @@ package v1alpha1
 
 import (
 	"fmt"
+	"time"
 
+	"github.com/keikoproj/upgrade-manager/controllers/common"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/util/intstr"
@@ -71,9 +73,23 @@ type RollingUpgradeStatus struct {
 	NodesProcessed      int    `json:"nodesProcessed,omitempty"`
 	TotalNodes          int    `json:"totalNodes,omitempty"`
 
-	Conditions []RollingUpgradeCondition `json:"conditions,omitempty"`
+	Conditions              []RollingUpgradeCondition `json:"conditions,omitempty"`
+	LastNodeTerminationTime metav1.Time               `json:"lastTerminationTime,omitempty"`
+	LastNodeDrainTime       metav1.Time               `json:"lastDrainTime,omitempty"`
+
+	Statistics     []*RollingUpgradeStatistics `json:"statistics,omitempty"`
+	LastBatchNodes []string                    `json:"lastBatchNodes,omitempty"`
+}
+
+// RollingUpgrade Statistics, includes summary(sum/count) from each step
+type RollingUpgradeStatistics struct {
+	StepName      RollingUpgradeStep `json:"stepName,omitempty"`
+	DurationSum   metav1.Duration    `json:"durationSum,omitempty"`
+	DurationCount int32              `json:"durationCount,omitempty"`
 }
 
+type RollingUpgradeStep string
+
 const (
 	// StatusRunning marks the CR to be running.
 	StatusRunning = "running"
@@ -81,14 +97,157 @@ const (
 	StatusComplete = "completed"
 	// StatusError marks the CR as errored out.
 	StatusError = "error"
+
+	NodeRotationTotal RollingUpgradeStep = "total"
+
+	NodeRotationKickoff          RollingUpgradeStep = "kickoff"
+	NodeRotationDesiredNodeReady RollingUpgradeStep = "desired_node_ready"
+	NodeRotationPredrainScript   RollingUpgradeStep = "predrain_script"
+	NodeRotationDrain            RollingUpgradeStep = "drain"
+	NodeRotationPostdrainScript  RollingUpgradeStep = "postdrain_script"
+	NodeRotationPostWait         RollingUpgradeStep = "post_wait"
+	NodeRotationTerminate        RollingUpgradeStep = "terminate"
+	NodeRotationPostTerminate    RollingUpgradeStep = "post_terminate"
+	NodeRotationCompleted        RollingUpgradeStep = "completed"
 )
 
+var NodeRotationStepOrders = map[RollingUpgradeStep]int{
+	NodeRotationKickoff:          10,
+	NodeRotationDesiredNodeReady: 20,
+	NodeRotationPredrainScript:   30,
+	NodeRotationDrain:            40,
+	NodeRotationPostdrainScript:  50,
+	NodeRotationPostWait:         60,
+	NodeRotationTerminate:        70,
+	NodeRotationPostTerminate:    80,
+	NodeRotationCompleted:        1000,
+}
+
 // RollingUpgradeCondition describes the state of the RollingUpgrade
 type RollingUpgradeCondition struct {
 	Type   UpgradeConditionType   `json:"type,omitempty"`
 	Status corev1.ConditionStatus `json:"status,omitempty"`
 }
 
+// RollingUpgrade Node step information
+type NodeStepDuration struct {
+	GroupName string             `json:"groupName,omitempty"`
+	NodeName  string             `json:"nodeName,omitempty"`
+	StepName  RollingUpgradeStep `json:"stepName,omitempty"`
+	Duration  metav1.Duration    `json:"duration,omitempty"`
+}
+
+// Node In-processing
+type NodeInProcessing struct {
+	NodeName         string             `json:"nodeName,omitempty"`
+	StepName         RollingUpgradeStep `json:"stepName,omitempty"`
+	UpgradeStartTime metav1.Time        `json:"upgradeStartTime,omitempty"`
+	StepStartTime    metav1.Time        `json:"stepStartTime,omitempty"`
+	StepEndTime      metav1.Time        `json:"stepEndTime,omitempty"`
+}
+
+// Update last batch nodes
+func (s *RollingUpgradeStatus) UpdateLastBatchNodes(batchNodes map[string]*NodeInProcessing) {
+	keys := make([]string, 0, len(batchNodes))
+	for k := range batchNodes {
+		keys = append(keys, k)
+	}
+	s.LastBatchNodes = keys
+}
+
+// Update Node Statistics
+func (s *RollingUpgradeStatus) UpdateStatistics(nodeSteps map[string][]NodeStepDuration) {
+	for _, v := range nodeSteps {
+		for _, step := range v {
+			s.AddNodeStepDuration(step)
+		}
+	}
+}
+
+// Add one step duration
+func (s *RollingUpgradeStatus) AddNodeStepDuration(nsd NodeStepDuration) {
+	// if step exists, add count and sum, otherwise append
+	for _, s := range s.Statistics {
+		if s.StepName == nsd.StepName {
+			s.DurationSum = metav1.Duration{
+				Duration: s.DurationSum.Duration + nsd.Duration.Duration,
+			}
+			s.DurationCount += 1
+			return
+		}
+	}
+	s.Statistics = append(s.Statistics, &RollingUpgradeStatistics{
+		StepName: nsd.StepName,
+		DurationSum: metav1.Duration{
+			Duration: nsd.Duration.Duration,
+		},
+		DurationCount: 1,
+	})
+}
+
+// Node turns onto step
+func (s *RollingUpgradeStatus) NodeStep(InProcessingNodes map[string]*NodeInProcessing,
+	nodeSteps map[string][]NodeStepDuration, groupName, nodeName string, stepName RollingUpgradeStep) {
+
+	var inProcessingNode *NodeInProcessing
+	if n, ok := InProcessingNodes[nodeName]; !ok {
+		inProcessingNode = &NodeInProcessing{
+			NodeName:         nodeName,
+			StepName:         stepName,
+			UpgradeStartTime: metav1.Now(),
+			StepStartTime:    metav1.Now(),
+		}
+		InProcessingNodes[nodeName] = inProcessingNode
+	} else {
+		inProcessingNode = n
+	}
+
+	inProcessingNode.StepEndTime = metav1.Now()
+	var duration = inProcessingNode.StepEndTime.Sub(inProcessingNode.StepStartTime.Time)
+	if stepName == NodeRotationCompleted {
+		//Add overall and remove the node from in-processing map
+		var total = inProcessingNode.StepEndTime.Sub(inProcessingNode.UpgradeStartTime.Time)
+		duration1 := s.ToStepDuration(groupName, nodeName, inProcessingNode.StepName, duration)
+		duration2 := s.ToStepDuration(groupName, nodeName, NodeRotationTotal, total)
+		s.addNodeStepDuration(nodeSteps, nodeName, duration1)
+		s.addNodeStepDuration(nodeSteps, nodeName, duration2)
+	} else if inProcessingNode.StepName != stepName { //Still same step
+		var oldOrder = NodeRotationStepOrders[inProcessingNode.StepName]
+		var newOrder = NodeRotationStepOrders[stepName]
+		if newOrder > oldOrder { //Make sure the steps running in order
+			stepDuration := s.ToStepDuration(groupName, nodeName, inProcessingNode.StepName, duration)
+			inProcessingNode.StepStartTime = metav1.Now()
+			inProcessingNode.StepName = stepName
+			s.addNodeStepDuration(nodeSteps, nodeName, stepDuration)
+		}
+	}
+}
+
+func (s *RollingUpgradeStatus) addNodeStepDuration(steps map[string][]NodeStepDuration, nodeName string, nsd NodeStepDuration) {
+	if stepDuration, ok := steps[nodeName]; !ok {
+		steps[nodeName] = []NodeStepDuration{
+			nsd,
+		}
+	} else {
+		stepDuration = append(stepDuration, nsd)
+		steps[nodeName] = stepDuration
+	}
+}
+
+// Add one step duration
+func (s *RollingUpgradeStatus) ToStepDuration(groupName, nodeName string, stepName RollingUpgradeStep, duration time.Duration) NodeStepDuration {
+	//Add to system level statistics
+	common.AddStepDuration(groupName, string(stepName), duration)
+	return NodeStepDuration{
+		GroupName: groupName,
+		NodeName:  nodeName,
+		StepName:  stepName,
+		Duration: metav1.Duration{
+			Duration: duration,
+		},
+	}
+}
+
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
 // +kubebuilder:resource:path=rollingupgrades,scope=Namespaced,shortName=ru
diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go
index 3f4d426..6446e34 100644
--- a/api/v1alpha1/zz_generated.deepcopy.go
+++ b/api/v1alpha1/zz_generated.deepcopy.go
@@ -23,6 +23,24 @@ import (
 	runtime "k8s.io/apimachinery/pkg/runtime"
 )
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *NodeInProcessing) DeepCopyInto(out *NodeInProcessing) {
+	*out = *in
+	in.UpgradeStartTime.DeepCopyInto(&out.UpgradeStartTime)
+	in.StepStartTime.DeepCopyInto(&out.StepStartTime)
+	in.StepEndTime.DeepCopyInto(&out.StepEndTime)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeInProcessing.
+func (in *NodeInProcessing) DeepCopy() *NodeInProcessing {
+	if in == nil {
+		return nil
+	}
+	out := new(NodeInProcessing)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *NodeReadinessGate) DeepCopyInto(out *NodeReadinessGate) {
 	*out = *in
@@ -45,6 +63,22 @@ func (in *NodeReadinessGate) DeepCopy() *NodeReadinessGate {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *NodeStepDuration) DeepCopyInto(out *NodeStepDuration) {
+	*out = *in
+	out.Duration = in.Duration
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeStepDuration.
+func (in *NodeStepDuration) DeepCopy() *NodeStepDuration {
+	if in == nil {
+		return nil
+	}
+	out := new(NodeStepDuration)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *PostDrainSpec) DeepCopyInto(out *PostDrainSpec) {
 	*out = *in
@@ -190,6 +224,22 @@ func (in *RollingUpgradeSpec) DeepCopy() *RollingUpgradeSpec {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *RollingUpgradeStatistics) DeepCopyInto(out *RollingUpgradeStatistics) {
+	*out = *in
+	out.DurationSum = in.DurationSum
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatistics.
+func (in *RollingUpgradeStatistics) DeepCopy() *RollingUpgradeStatistics {
+	if in == nil {
+		return nil
+	}
+	out := new(RollingUpgradeStatistics)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) {
 	*out = *in
@@ -198,6 +248,24 @@ func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) {
 		*out = make([]RollingUpgradeCondition, len(*in))
 		copy(*out, *in)
 	}
+	in.LastNodeTerminationTime.DeepCopyInto(&out.LastNodeTerminationTime)
+	in.LastNodeDrainTime.DeepCopyInto(&out.LastNodeDrainTime)
+	if in.Statistics != nil {
+		in, out := &in.Statistics, &out.Statistics
+		*out = make([]*RollingUpgradeStatistics, len(*in))
+		for i := range *in {
+			if (*in)[i] != nil {
+				in, out := &(*in)[i], &(*out)[i]
+				*out = new(RollingUpgradeStatistics)
+				**out = **in
+			}
+		}
+	}
+	if in.LastBatchNodes != nil {
+		in, out := &in.LastBatchNodes, &out.LastBatchNodes
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatus.
diff --git a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml
index 04fa959..358bc17 100644
--- a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml
+++ b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml
@@ -147,10 +147,34 @@ spec:
               type: string
             endTime:
               type: string
+            lastBatchNodes:
+              items:
+                type: string
+              type: array
+            lastDrainTime:
+              format: date-time
+              type: string
+            lastTerminationTime:
+              format: date-time
+              type: string
             nodesProcessed:
               type: integer
             startTime:
               type: string
+            statistics:
+              items:
+                description: RollingUpgrade Statistics, includes summary(sum/count)
+                  from each step
+                properties:
+                  durationCount:
+                    format: int32
+                    type: integer
+                  durationSum:
+                    type: string
+                  stepName:
+                    type: string
+                type: object
+              type: array
             totalNodes:
               type: integer
             totalProcessingTime:
diff --git a/controllers/common/metrics.go b/controllers/common/metrics.go
new file mode 100644
index 0000000..eb74380
--- /dev/null
+++ b/controllers/common/metrics.go
@@ -0,0 +1,75 @@
+package common
+
+import (
+	"reflect"
+	"strings"
+	"time"
+
+	"github.com/keikoproj/upgrade-manager/pkg/log"
+	"github.com/prometheus/client_golang/prometheus"
+	"sigs.k8s.io/controller-runtime/pkg/metrics"
+)
+
+//All cluster level node upgrade statistics
+
+var nodeRotationTotal = prometheus.NewHistogram(
+	prometheus.HistogramOpts{
+		Namespace: "node",
+		Name:      "rotation_total_seconds",
+		Help:      "Node rotation total",
+		Buckets: []float64{
+			10.0,
+			30.0,
+			60.0,
+			90.0,
+			120.0,
+			180.0,
+			300.0,
+			600.0,
+			900.0,
+		},
+	})
+
+var stepSummaries = make(map[string]map[string]prometheus.Summary)
+
+func InitMetrics() {
+	metrics.Registry.MustRegister(nodeRotationTotal)
+}
+
+// Add rolling update step duration when the step is completed
+func AddStepDuration(groupName string, stepName string, duration time.Duration) {
+	if strings.EqualFold(stepName, "total") { //Histogram
+		nodeRotationTotal.Observe(duration.Seconds())
+	} else { //Summary
+		var steps map[string]prometheus.Summary
+		if m, ok := stepSummaries[groupName]; !ok {
+			steps = make(map[string]prometheus.Summary)
+			stepSummaries[groupName] = steps
+		} else {
+			steps = m
+		}
+
+		var summary prometheus.Summary
+		if s, ok := steps[stepName]; !ok {
+			summary = prometheus.NewSummary(
+				prometheus.SummaryOpts{
+					Namespace:   "node",
+					Name:        stepName + "_seconds",
+					Help:        "Summary for node " + stepName,
+					ConstLabels: prometheus.Labels{"group": groupName},
+				})
+			err := metrics.Registry.Register(summary)
+			if err != nil {
+				if reflect.TypeOf(err).String() == "prometheus.AlreadyRegisteredError" {
+					log.Warnf("summary was registered again, group: %s, step: %s", groupName, stepName)
+				} else {
+					log.Errorf("register summary error, group: %s, step: %s, %v", groupName, stepName, err)
+				}
+			}
+			steps[stepName] = summary
+		} else {
+			summary = s
+		}
+		summary.Observe(duration.Seconds())
+	}
+}
diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go
index b647d97..1a0d43a 100644
--- a/controllers/rollingupgrade_controller.go
+++ b/controllers/rollingupgrade_controller.go
@@ -43,6 +43,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/controller"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 
+	"github.com/keikoproj/upgrade-manager/api/v1alpha1"
 	upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1"
 )
 
@@ -148,7 +149,11 @@ func (r *RollingUpgradeReconciler) preDrainHelper(instanceID, nodeName string, r
 
 // Operates on any scripts that were provided after the draining of the node.
 // kubeCtlCall is provided as an argument to decouple the method from the actual kubectl call
-func (r *RollingUpgradeReconciler) postDrainHelper(instanceID, nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade) error {
+func (r *RollingUpgradeReconciler) postDrainHelper(instanceID,
+	nodeName string,
+	ruObj *upgrademgrv1alpha1.RollingUpgrade,
+	nodeSteps map[string][]v1alpha1.NodeStepDuration,
+	inProcessingNodes map[string]*v1alpha1.NodeInProcessing) error {
 	err := r.ScriptRunner.PostDrain(instanceID, nodeName, ruObj)
 	if err != nil {
 		return err
@@ -157,6 +162,8 @@ func (r *RollingUpgradeReconciler) postDrainHelper(instanceID, nodeName string,
 	r.info(ruObj, "Waiting for postDrainDelay", "postDrainDelay", ruObj.Spec.PostDrainDelaySeconds)
 	time.Sleep(time.Duration(ruObj.Spec.PostDrainDelaySeconds) * time.Second)
 
+	ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait)
+
 	return r.ScriptRunner.PostWait(instanceID, nodeName, ruObj)
 
 }
@@ -166,7 +173,12 @@ func (r *RollingUpgradeReconciler) postDrainHelper(instanceID, nodeName string,
 func (r *RollingUpgradeReconciler) DrainNode(ruObj *upgrademgrv1alpha1.RollingUpgrade,
 	nodeName string,
 	instanceID string,
-	drainTimeout int) error {
+	drainTimeout int,
+	nodeSteps map[string][]v1alpha1.NodeStepDuration,
+	inProcessingNodes map[string]*v1alpha1.NodeInProcessing) error {
+
+	ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript)
+
 	// Running kubectl drain node.
 	err := r.preDrainHelper(instanceID, nodeName, ruObj)
 	if err != nil {
@@ -188,6 +200,8 @@ func (r *RollingUpgradeReconciler) DrainNode(ruObj *upgrademgrv1alpha1.RollingUp
 		r.info(ruObj, "Skipped creating context with timeout.", "drainTimeout", drainTimeout)
 	}
 
+	ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain)
+
 	r.info(ruObj, "Invoking kubectl drain for the node", "nodeName", nodeName)
 	go r.CallKubectlDrain(nodeName, ruObj, errChan)
 
@@ -206,7 +220,9 @@ func (r *RollingUpgradeReconciler) DrainNode(ruObj *upgrademgrv1alpha1.RollingUp
 		r.info(ruObj, "Kubectl drain completed for node", "nodeName", nodeName)
 	}
 
-	return r.postDrainHelper(instanceID, nodeName, ruObj)
+	ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript)
+
+	return r.postDrainHelper(instanceID, nodeName, ruObj, nodeSteps, inProcessingNodes)
 }
 
 // CallKubectlDrain runs the "kubectl drain" for a given node
@@ -365,12 +381,18 @@ func (r *RollingUpgradeReconciler) SetStandby(ruObj *upgrademgrv1alpha1.RollingU
 }
 
 // TerminateNode actually terminates the given node.
-func (r *RollingUpgradeReconciler) TerminateNode(ruObj *upgrademgrv1alpha1.RollingUpgrade, instanceID string, nodeName string) error {
+func (r *RollingUpgradeReconciler) TerminateNode(ruObj *upgrademgrv1alpha1.RollingUpgrade,
+	instanceID string,
+	nodeName string,
+	nodeSteps map[string][]v1alpha1.NodeStepDuration,
+	inProcessingNodes map[string]*v1alpha1.NodeInProcessing) error {
 
 	input := &autoscaling.TerminateInstanceInAutoScalingGroupInput{
 		InstanceId:                     aws.String(instanceID),
 		ShouldDecrementDesiredCapacity: aws.Bool(false),
 	}
+	ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate)
+
 	var err error
 	var ieb *iebackoff.IEBackoff
 	for ieb, err = iebackoff.NewIEBackoff(WaiterMaxDelay, WaiterMinDelay, 0.5, WaiterMaxAttempts); err == nil; err = ieb.Next() {
@@ -400,6 +422,7 @@ func (r *RollingUpgradeReconciler) TerminateNode(ruObj *upgrademgrv1alpha1.Rolli
 	r.info(ruObj, "Instance terminated.", "instanceID", instanceID)
 	r.info(ruObj, "starting post termination sleep", "instanceID", instanceID, "nodeIntervalSeconds", ruObj.Spec.NodeIntervalSeconds)
 	time.Sleep(time.Duration(ruObj.Spec.NodeIntervalSeconds) * time.Second)
+	ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate)
 	return r.ScriptRunner.PostTerminate(instanceID, nodeName, ruObj)
 }
 
@@ -915,6 +938,10 @@ func (r *RollingUpgradeReconciler) UpdateInstances(ctx *context.Context,
 		return nil
 	}
 
+	//A map to retain the steps for multiple nodes
+	nodeSteps := make(map[string][]v1alpha1.NodeStepDuration)
+	inProcessingNodes := make(map[string]*v1alpha1.NodeInProcessing)
+
 	ch := make(chan error)
 
 	for _, instance := range instances {
@@ -925,7 +952,7 @@ func (r *RollingUpgradeReconciler) UpdateInstances(ctx *context.Context,
 			"strategy": string(ruObj.Spec.Strategy.Type),
 			"msg":      fmt.Sprintf("Started Updating Instance %s, in AZ: %s", *instance.InstanceId, *instance.AvailabilityZone),
 		})
-		go r.UpdateInstance(ctx, ruObj, instance, launchDefinition, ch)
+		go r.UpdateInstance(ctx, ruObj, instance, launchDefinition, ch, nodeSteps, inProcessingNodes)
 	}
 
 	// wait for upgrades to complete
@@ -953,6 +980,9 @@ func (r *RollingUpgradeReconciler) UpdateInstances(ctx *context.Context,
 		}
 	}
 
+	ruObj.Status.UpdateStatistics(nodeSteps)
+	ruObj.Status.UpdateLastBatchNodes(inProcessingNodes)
+
 	if len(instanceUpdateErrors) > 0 {
 		return NewUpdateInstancesError(instanceUpdateErrors)
 	}
@@ -962,7 +992,9 @@ func (r *RollingUpgradeReconciler) UpdateInstances(ctx *context.Context,
 func (r *RollingUpgradeReconciler) UpdateInstanceEager(
 	ruObj *upgrademgrv1alpha1.RollingUpgrade,
 	nodeName,
-	targetInstanceID string) error {
+	targetInstanceID string,
+	nodeSteps map[string][]v1alpha1.NodeStepDuration,
+	inProcessingNodes map[string]*v1alpha1.NodeInProcessing) error {
 
 	// Set instance to standby
 	err := r.SetStandby(ruObj, targetInstanceID)
@@ -970,6 +1002,8 @@ func (r *RollingUpgradeReconciler) UpdateInstanceEager(
 		return err
 	}
 
+	ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady)
+
 	// Wait for new instance to be created
 	err = r.WaitForDesiredInstances(ruObj)
 	if err != nil {
@@ -983,27 +1017,31 @@ func (r *RollingUpgradeReconciler) UpdateInstanceEager(
 	}
 
 	// Drain and wait for draining node.
-	return r.DrainTerminate(ruObj, nodeName, targetInstanceID)
+	return r.DrainTerminate(ruObj, nodeName, targetInstanceID, nodeSteps, inProcessingNodes)
 }
 
 func (r *RollingUpgradeReconciler) DrainTerminate(
 	ruObj *upgrademgrv1alpha1.RollingUpgrade,
 	nodeName,
-	targetInstanceID string) error {
+	targetInstanceID string,
+	nodeSteps map[string][]v1alpha1.NodeStepDuration,
+	inProcessingNodes map[string]*v1alpha1.NodeInProcessing) error {
 
 	// Drain and wait for draining node.
 	if nodeName != "" {
-		if err := r.DrainNode(ruObj, nodeName, targetInstanceID, ruObj.Spec.Strategy.DrainTimeout); err != nil {
+		if err := r.DrainNode(ruObj, nodeName, targetInstanceID, ruObj.Spec.Strategy.DrainTimeout, nodeSteps, inProcessingNodes); err != nil {
 			return err
 		}
 	}
 
 	// Terminate instance.
-	err := r.TerminateNode(ruObj, targetInstanceID, nodeName)
+	err := r.TerminateNode(ruObj, targetInstanceID, nodeName, nodeSteps, inProcessingNodes)
 	if err != nil {
 		return err
 	}
 
+	ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted)
+
 	return nil
 }
 
@@ -1012,7 +1050,9 @@ func (r *RollingUpgradeReconciler) UpdateInstance(ctx *context.Context,
 	ruObj *upgrademgrv1alpha1.RollingUpgrade,
 	i *autoscaling.Instance,
 	launchDefinition *launchDefinition,
-	ch chan error) {
+	ch chan error,
+	nodeSteps map[string][]v1alpha1.NodeStepDuration,
+	inProcessingNodes map[string]*v1alpha1.NodeInProcessing) {
 	targetInstanceID := aws.StringValue(i.InstanceId)
 	// If an instance was marked as "in-progress" in ClusterState, it has to be marked
 	// completed so that it can get considered again in a subsequent rollup CR.
@@ -1056,10 +1096,14 @@ func (r *RollingUpgradeReconciler) UpdateInstance(ctx *context.Context,
 	mode := ruObj.Spec.Strategy.Mode.String()
 	if strings.ToLower(mode) == upgrademgrv1alpha1.UpdateStrategyModeEager.String() {
 		r.info(ruObj, "starting replacement with eager mode", "mode", mode)
-		err = r.UpdateInstanceEager(ruObj, nodeName, targetInstanceID)
+		//Add statistics
+		ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff)
+		err = r.UpdateInstanceEager(ruObj, nodeName, targetInstanceID, nodeSteps, inProcessingNodes)
 	} else if strings.ToLower(mode) == upgrademgrv1alpha1.UpdateStrategyModeLazy.String() {
 		r.info(ruObj, "starting replacement with lazy mode", "mode", mode)
-		err = r.DrainTerminate(ruObj, nodeName, targetInstanceID)
+		//Add statistics
+		ruObj.Status.NodeStep(inProcessingNodes, nodeSteps, ruObj.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff)
+		err = r.DrainTerminate(ruObj, nodeName, targetInstanceID, nodeSteps, inProcessingNodes)
 	} else {
 		err = fmt.Errorf("%s: unhandled strategy mode: %s", ruObj.NamespacedName(), mode)
 	}
diff --git a/deploy/rolling-upgrade-controller-deploy.yaml b/deploy/rolling-upgrade-controller-deploy.yaml
index 00233d7..a9ebb74 100644
--- a/deploy/rolling-upgrade-controller-deploy.yaml
+++ b/deploy/rolling-upgrade-controller-deploy.yaml
@@ -51,6 +51,9 @@ spec:
           requests:
             cpu: 100m
             memory: 300Mi
+        ports:
+          - containerPort: 8080
+            name: metrics
         terminationMessagePath: /dev/termination-log
         terminationMessagePolicy: File
       dnsPolicy: ClusterFirst
@@ -65,3 +68,19 @@ spec:
       serviceAccount: rolling-upgrade-sa
       serviceAccountName: rolling-upgrade-sa
       terminationGracePeriodSeconds: 30
+---
+kind: Service
+apiVersion: v1
+metadata:
+  labels:
+    iks-metric: http-metrics
+  name: rolling-upgrade-controller-service
+  namespace: kube-system
+spec:
+  ports:
+    - name: iks-metric
+      port: 8080
+      protocol: TCP
+      targetPort: 8080
+  selector:
+    app: rolling-upgrade-controller
\ No newline at end of file
diff --git a/go.mod b/go.mod
index 34780d7..b438544 100644
--- a/go.mod
+++ b/go.mod
@@ -11,6 +11,7 @@ require (
 	github.com/keikoproj/kubedog v0.0.1
 	github.com/onsi/ginkgo v1.14.2
 	github.com/onsi/gomega v1.10.4
+	github.com/prometheus/client_golang v1.0.0
 	github.com/sirupsen/logrus v1.6.0
 	go.uber.org/zap v1.16.0
 	golang.org/x/net v0.0.0-20201216054612-986b41b23924
diff --git a/main.go b/main.go
index f08b940..ba5e55d 100644
--- a/main.go
+++ b/main.go
@@ -39,6 +39,7 @@ import (
 
 	upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1"
 	"github.com/keikoproj/upgrade-manager/controllers"
+	"github.com/keikoproj/upgrade-manager/controllers/common"
 	"github.com/keikoproj/upgrade-manager/pkg/log"
 	// +kubebuilder:scaffold:imports
 )
@@ -63,6 +64,7 @@ func init() {
 		panic(err)
 	}
 	// +kubebuilder:scaffold:scheme
+	common.InitMetrics()
 }
 
 func main() {
@@ -161,6 +163,8 @@ func main() {
 	}
 	// +kubebuilder:scaffold:builder
 
+	setupLog.Info("registering prometheus")
+
 	setupLog.Info("starting manager")
 	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
 		setupLog.Error(err, "problem running manager")


Test patch:
diff --git a/api/v1alpha1/rollingupgrade_types_test.go b/api/v1alpha1/rollingupgrade_types_test.go
index c6c737c..74396c1 100644
--- a/api/v1alpha1/rollingupgrade_types_test.go
+++ b/api/v1alpha1/rollingupgrade_types_test.go
@@ -16,7 +16,10 @@ limitations under the License.
 package v1alpha1
 
 import (
+	"testing"
+
 	. "github.com/onsi/ginkgo"
+	"github.com/onsi/gomega"
 	. "github.com/onsi/gomega"
 
 	"golang.org/x/net/context"
@@ -81,3 +84,54 @@ var _ = Describe("RollingUpgrade", func() {
 	})
 
 })
+
+// Test
+func TestNodeTurnsOntoStep(t *testing.T) {
+	g := gomega.NewGomegaWithT(t)
+
+	r := &RollingUpgradeStatus{}
+	//A map to retain the steps for multiple nodes
+	nodeSteps := make(map[string][]NodeStepDuration)
+	inProcessingNodes := make(map[string]*NodeInProcessing)
+
+	r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationKickoff)
+
+	g.Expect(inProcessingNodes).NotTo(gomega.BeNil())
+	g.Expect(nodeSteps["node-1"]).To(gomega.BeNil())
+
+	r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationDesiredNodeReady)
+
+	g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(1))
+	g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(NodeRotationKickoff))
+
+	//Retry desired_node_ready
+	r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationDesiredNodeReady)
+	g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(1))
+	g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(NodeRotationKickoff))
+
+	//Retry desired_node_ready again
+	r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationDesiredNodeReady)
+	g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(1))
+	g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(NodeRotationKickoff))
+
+	//Completed
+	r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationCompleted)
+	g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(3))
+	g.Expect(nodeSteps["node-1"][1].StepName).To(gomega.Equal(NodeRotationDesiredNodeReady))
+	g.Expect(nodeSteps["node-1"][2].StepName).To(gomega.Equal(NodeRotationTotal))
+
+	//Second node
+	r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-2", NodeRotationKickoff)
+	g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(3))
+
+	r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-2", NodeRotationDesiredNodeReady)
+	g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(3))
+
+	r.UpdateLastBatchNodes(inProcessingNodes)
+	g.Expect(len(r.LastBatchNodes)).To(gomega.Equal(2))
+
+	r.UpdateStatistics(nodeSteps)
+	g.Expect(r.Statistics).ToNot(gomega.BeEmpty())
+	g.Expect(len(r.Statistics)).To(gomega.Equal(3))
+
+}
diff --git a/controllers/common/metrics_test.go b/controllers/common/metrics_test.go
new file mode 100644
index 0000000..c6b5b7f
--- /dev/null
+++ b/controllers/common/metrics_test.go
@@ -0,0 +1,30 @@
+package common
+
+import (
+	"testing"
+
+	"github.com/onsi/gomega"
+)
+
+func TestAddRollingUpgradeStepDuration(t *testing.T) {
+	g := gomega.NewGomegaWithT(t)
+
+	g.Expect(stepSummaries["test-asg"]).To(gomega.BeNil())
+	AddStepDuration("test-asg", "kickoff", 1)
+
+	g.Expect(stepSummaries["test-asg"]).NotTo(gomega.BeNil())
+	g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil())
+
+	//Test duplicate
+	AddStepDuration("test-asg", "kickoff", 1)
+	g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil())
+
+	//Test duplicate
+	delete(stepSummaries["test-asg"], "kickoff")
+	AddStepDuration("test-asg", "kickoff", 1)
+	g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil())
+
+	//Test total
+	AddStepDuration("test-asg", "total", 1)
+	g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil())
+}
diff --git a/controllers/rollingupgrade_controller_test.go b/controllers/rollingupgrade_controller_test.go
index 892d5d1..6c7cc72 100644
--- a/controllers/rollingupgrade_controller_test.go
+++ b/controllers/rollingupgrade_controller_test.go
@@ -38,6 +38,9 @@ import (
 	upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1"
 )
 
+var nodeSteps = make(map[string][]upgrademgrv1alpha1.NodeStepDuration)
+var inProcessingNodes = make(map[string]*upgrademgrv1alpha1.NodeInProcessing)
+
 func TestMain(m *testing.M) {
 	testEnv = &envtest.Environment{
 		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
@@ -135,7 +138,7 @@ func TestPostDrainHelperPostDrainScriptSuccess(t *testing.T) {
 
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
-	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj)
+	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj, nodeSteps, inProcessingNodes)
 
 	g.Expect(err).To(gomega.BeNil())
 }
@@ -151,7 +154,7 @@ func TestPostDrainHelperPostDrainScriptError(t *testing.T) {
 
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
-	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj)
+	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj, nodeSteps, inProcessingNodes)
 
 	g.Expect(err).To(gomega.Not(gomega.BeNil()))
 
@@ -174,7 +177,7 @@ func TestPostDrainHelperPostDrainScriptErrorWithIgnoreDrainFailures(t *testing.T
 
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
-	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj)
+	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj, nodeSteps, inProcessingNodes)
 
 	g.Expect(err).To(gomega.Not(gomega.BeNil()))
 
@@ -195,7 +198,7 @@ func TestPostDrainHelperPostDrainWaitScriptSuccess(t *testing.T) {
 
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
-	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj)
+	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj, nodeSteps, inProcessingNodes)
 
 	g.Expect(err).To(gomega.BeNil())
 }
@@ -212,7 +215,7 @@ func TestPostDrainHelperPostDrainWaitScriptError(t *testing.T) {
 
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
-	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj)
+	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj, nodeSteps, inProcessingNodes)
 
 	g.Expect(err).To(gomega.Not(gomega.BeNil()))
 
@@ -236,7 +239,7 @@ func TestPostDrainHelperPostDrainWaitScriptErrorWithIgnoreDrainFailures(t *testi
 
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
-	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj)
+	err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj, nodeSteps, inProcessingNodes)
 
 	g.Expect(err).To(gomega.Not(gomega.BeNil()))
 
@@ -258,7 +261,7 @@ func TestDrainNodeSuccess(t *testing.T) {
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
 
-	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout)
+	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout, nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.BeNil())
 }
 
@@ -272,7 +275,7 @@ func TestDrainNodePreDrainError(t *testing.T) {
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
 
-	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout)
+	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout, nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.Not(gomega.BeNil()))
 }
 
@@ -286,7 +289,7 @@ func TestDrainNodePostDrainScriptError(t *testing.T) {
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
 
-	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout)
+	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout, nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.Not(gomega.BeNil()))
 }
 
@@ -300,7 +303,7 @@ func TestDrainNodePostDrainWaitScriptError(t *testing.T) {
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
 
-	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout)
+	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout, nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.Not(gomega.BeNil()))
 }
 
@@ -318,7 +321,7 @@ func TestDrainNodePostDrainFailureToDrainNotFound(t *testing.T) {
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
 
-	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout)
+	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout, nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.BeNil())
 }
 
@@ -338,7 +341,7 @@ func TestDrainNodePostDrainFailureToDrain(t *testing.T) {
 	rcRollingUpgrade := createReconciler()
 	rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall
 
-	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout)
+	err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout, nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.Not(gomega.BeNil()))
 }
 
@@ -483,7 +486,7 @@ func TestTerminateNodeSuccess(t *testing.T) {
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
 
-	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "")
+	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "", nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.BeNil())
 }
 
@@ -503,7 +506,7 @@ func TestTerminateNodeErrorNotFound(t *testing.T) {
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
 
-	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "")
+	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "", nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.BeNil())
 }
 
@@ -535,7 +538,7 @@ func TestTerminateNodeErrorScalingActivityInProgressWithRetry(t *testing.T) {
 			awsErr:    nil,
 		}
 	}()
-	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "")
+	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "", nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.BeNil())
 }
 
@@ -554,7 +557,7 @@ func TestTerminateNodeErrorScalingActivityInProgress(t *testing.T) {
 		EC2Client:    MockEC2{},
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
-	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "")
+	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "", nodeSteps, inProcessingNodes)
 	g.Expect(err.Error()).To(gomega.ContainSubstring("no more retries left"))
 }
 
@@ -574,7 +577,7 @@ func TestTerminateNodeErrorResourceContention(t *testing.T) {
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
 
-	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "")
+	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "", nodeSteps, inProcessingNodes)
 	g.Expect(err.Error()).To(gomega.ContainSubstring("no more retries left"))
 }
 
@@ -594,7 +597,7 @@ func TestTerminateNodeErrorOtherError(t *testing.T) {
 		EC2Client:    MockEC2{},
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
-	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "")
+	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "", nodeSteps, inProcessingNodes)
 	g.Expect(err.Error()).To(gomega.ContainSubstring("some error"))
 }
 
@@ -612,7 +615,7 @@ func TestTerminateNodePostTerminateScriptSuccess(t *testing.T) {
 		EC2Client:    MockEC2{},
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
-	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "")
+	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "", nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.BeNil())
 }
 
@@ -630,7 +633,7 @@ func TestTerminateNodePostTerminateScriptErrorNotFoundFromServer(t *testing.T) {
 		EC2Client:    MockEC2{},
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
-	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "")
+	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "", nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.BeNil())
 }
 
@@ -648,7 +651,7 @@ func TestTerminateNodePostTerminateScriptErrorOtherError(t *testing.T) {
 		EC2Client:    MockEC2{},
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
-	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "")
+	err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "", nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.Not(gomega.BeNil()))
 	g.Expect(err.Error()).To(gomega.ContainSubstring("Failed to run postTerminate script: "))
 }
@@ -1500,6 +1503,8 @@ func TestUpdateInstances(t *testing.T) {
 	err = rcRollingUpgrade.UpdateInstances(&ctx,
 		ruObj, mockAsg.Instances, &launchDefinition{launchConfigurationName: &lcName})
 	g.Expect(err).ShouldNot(gomega.HaveOccurred())
+	g.Expect(ruObj.Status.Statistics).ShouldNot(gomega.BeEmpty())
+	g.Expect(ruObj.Status.LastBatchNodes).ShouldNot(gomega.BeEmpty())
 }
 
 func TestUpdateInstancesError(t *testing.T) {
@@ -2735,11 +2740,11 @@ func TestDrainNodeTerminateTerminatesWhenIgnoreDrainFailuresSet(t *testing.T) {
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
 
-	err := rcRollingUpgrade.DrainTerminate(ruObj, mockNode, mockNode)
+	err := rcRollingUpgrade.DrainTerminate(ruObj, mockNode, mockNode, nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.BeNil()) // don't expect errors.
 
 	// nodeName is empty when node isn't part of the cluster. It must skip drain and terminate.
-	err = rcRollingUpgrade.DrainTerminate(ruObj, "", mockNode)
+	err = rcRollingUpgrade.DrainTerminate(ruObj, "", mockNode, nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.BeNil()) // don't expect errors.
 
 }
@@ -2771,7 +2776,7 @@ func TestPreDrainFailureWhenIgnoreDrainFailuresSet(t *testing.T) {
 		ScriptRunner: NewScriptRunner(log2.NullLogger{}),
 	}
 
-	err := rcRollingUpgrade.DrainTerminate(ruObj, mockNode, mockNode)
+	err := rcRollingUpgrade.DrainTerminate(ruObj, mockNode, mockNode, nodeSteps, inProcessingNodes)
 	g.Expect(err).To(gomega.Not(gomega.BeNil())) // expect error, as the predrain failure shouldn't be masked when IgnoreDrainFailures is set.
 }
 func TestUpdateInstancesNotExists(t *testing.T) {
@@ -2821,7 +2826,7 @@ func TestUpdateInstancesNotExists(t *testing.T) {
 	instChan := make(chan error)
 	mockInstanceName1 := "foo1"
 	instance1 := autoscaling.Instance{InstanceId: &mockInstanceName1, AvailabilityZone: &az}
-	go rcRollingUpgrade.UpdateInstance(&ctx, ruObj, &instance1, &launchDefinition{launchConfigurationName: &lcName}, instChan)
+	go rcRollingUpgrade.UpdateInstance(&ctx, ruObj, &instance1, &launchDefinition{launchConfigurationName: &lcName}, instChan, nodeSteps, inProcessingNodes)
 	processCount := 0
 	select {
 	case <-ctx.Done():