Repository: keikoproj/instance-manager
Pull request #326 (2021-07-26 21:52:39)

feat: add support for calculating max pods when prefix assignment is …
…enabled

AWS recently released 1.9.0 of the AWS VPC K8S CNI, which adds support for prefix assignment. This PR 
allows users who are using prefix assignment to have accurate max-pods calculations through the use of a new annotation on IGs.

https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.9.0

Signed-off-by: Jonah Back <jonah@jonahback.com>

codecov[bot]: # [Codecov](https://codecov.io/gh/keikoproj/instance-manager/pull/326?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj) Report
> Merging [#326](https://codecov.io/gh/keikoproj/instance-manager/pull/326?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj) (69c78b6) into [master](https://codecov.io/gh/keikoproj/instance-manager/commit/2d0866e0215200ee26de69f8e604ba91131a32a3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj) (2d0866e) will **increase** coverage by `0.06%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/keikoproj/instance-manager/pull/326/graphs/tree.svg?width=650&height=150&src=pr&token=IJbjmSBliL&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj)](https://codecov.io/gh/keikoproj/instance-manager/pull/326?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj)

```diff
@@            Coverage Diff             @@
##           master     #326      +/-   ##
==========================================
+ Coverage   55.94%   56.01%   +0.06%     
==========================================
  Files          27       27              
  Lines        4020     4026       +6     
==========================================
+ Hits         2249     2255       +6     
  Misses       1629     1629              
  Partials      142      142              
```


| [Impacted Files](https://codecov.io/gh/keikoproj/instance-manager/pull/326?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj) | Coverage Δ | |
|---|---|---|
| [controllers/provisioners/eks/eks.go](https://codecov.io/gh/keikoproj/instance-manager/pull/326/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj#diff-Y29udHJvbGxlcnMvcHJvdmlzaW9uZXJzL2Vrcy9la3MuZ28=) | `89.13% <ø> (ø)` | |
| [controllers/provisioners/eks/helpers.go](https://codecov.io/gh/keikoproj/instance-manager/pull/326/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj#diff-Y29udHJvbGxlcnMvcHJvdmlzaW9uZXJzL2Vrcy9oZWxwZXJzLmdv) | `91.16% <100.00%> (+0.07%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/keikoproj/instance-manager/pull/326?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/keikoproj/instance-manager/pull/326?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj). Last update [2d0866e...69c78b6](https://codecov.io/gh/keikoproj/instance-manager/pull/326?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=keikoproj).


Patch:
diff --git a/controllers/provisioners/eks/eks.go b/controllers/provisioners/eks/eks.go
index 311ce53..bbd68b0 100644
--- a/controllers/provisioners/eks/eks.go
+++ b/controllers/provisioners/eks/eks.go
@@ -31,14 +31,15 @@ import (
 )
 
 const (
-	ProvisionerName                     = "eks"
-	defaultLaunchConfigurationRetention = 2
-	OverrideDefaultLabelsAnnotation     = "instancemgr.keikoproj.io/default-labels"
-	IRSAEnabledAnnotation               = "instancemgr.keikoproj.io/irsa-enabled"
-	OsFamilyAnnotation                  = "instancemgr.keikoproj.io/os-family"
-	ClusterAutoscalerEnabledAnnotation  = "instancemgr.keikoproj.io/cluster-autoscaler-enabled"
-	CustomNetworkingEnabledAnnotation   = "instancemgr.keikoproj.io/custom-networking-enabled"
-	CustomNetworkingHostPodsAnnotation  = "instancemgr.keikoproj.io/custom-networking-host-pods"
+	ProvisionerName                                   = "eks"
+	defaultLaunchConfigurationRetention               = 2
+	OverrideDefaultLabelsAnnotation                   = "instancemgr.keikoproj.io/default-labels"
+	IRSAEnabledAnnotation                             = "instancemgr.keikoproj.io/irsa-enabled"
+	OsFamilyAnnotation                                = "instancemgr.keikoproj.io/os-family"
+	ClusterAutoscalerEnabledAnnotation                = "instancemgr.keikoproj.io/cluster-autoscaler-enabled"
+	CustomNetworkingEnabledAnnotation                 = "instancemgr.keikoproj.io/custom-networking-enabled"
+	CustomNetworkingHostPodsAnnotation                = "instancemgr.keikoproj.io/custom-networking-host-pods"
+	CustomNetworkingPrefixAssignmentEnabledAnnotation = "instancemgr.keikoproj.io/custom-networking-prefix-assignment-enabled"
 
 	OsFamilyWindows      = "windows"
 	OsFamilyBottleRocket = "bottlerocket"
diff --git a/controllers/provisioners/eks/helpers.go b/controllers/provisioners/eks/helpers.go
index 15c87d5..ea524e7 100644
--- a/controllers/provisioners/eks/helpers.go
+++ b/controllers/provisioners/eks/helpers.go
@@ -502,8 +502,17 @@ func (ctx *EksInstanceGroupContext) GetComputedBootstrapOptions() *v1alpha1.Boot
 		}
 
 		instanceTypeNetworkInfo := awsprovider.GetInstanceTypeNetworkInfo(state.GetInstanceTypeInfo(), configuration.InstanceType)
-		maxPods := (*instanceTypeNetworkInfo.MaximumNetworkInterfaces-1)*
-			(*instanceTypeNetworkInfo.Ipv4AddressesPerInterface-1) + hostNetworkPods
+		var prefixAssignmentEnabled = instanceGroup.GetAnnotations()[CustomNetworkingPrefixAssignmentEnabledAnnotation] == "true"
+		var maxPods int64 = 0
+
+		var enis = aws.Int64Value(instanceTypeNetworkInfo.MaximumNetworkInterfaces)-1 //Primary interface is not used for pod networking when custom networking is enabled
+		var ipsPerInterface int64 = 1
+		if prefixAssignmentEnabled {
+			ipsPerInterface = 16 //Number of ips in a /28 block
+		}
+
+		maxPods = enis * ((aws.Int64Value(instanceTypeNetworkInfo.Ipv4AddressesPerInterface)-1) * ipsPerInterface)+ hostNetworkPods
+
 		if configuration.BootstrapOptions == nil {
 			return &v1alpha1.BootstrapOptions{
 				MaxPods: maxPods,
diff --git a/docs/EKS.md b/docs/EKS.md
index d69dad4..9fac3ce 100644
--- a/docs/EKS.md
+++ b/docs/EKS.md
@@ -655,4 +655,5 @@ The following operators are supported:
 |instancemgr.keikoproj.io/os-family|InstanceGroup|either "windows", "bottlerocket", or "amazonlinux2" (default)|this is required if you are running a windows or bottlerocket based AMI, by default the controller will try to bootstrap an amazonlinux2 AMI|
 |instancemgr.keikoproj.io/default-labels|InstanceGroup|comma-seprarated key-value string e.g. "label1=value1,label2=value2"|allows overriding the default node labels added by the controller, by default the role label is added depending on the cluster version|
 |instancemgr.keikoproj.io/custom-networking-enabled|InstanceGroup|"true"|setting this annotation to true will automatically calculate the correct setting for max pods and pass it to the kubelet|
+|instancemgr.keikoproj.io/custom-networking-prefix-assignment-enabled|InstanceGroup|"true"|setting this annotation to true will change the max pod calculations to reflect the pod density supported by vpc prefix assignment. Supported in AWS VPC CNI versions 1.9.0 and above - see [AWS VPC CNI 1.9.0](https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.9.0) for more information.|
 |instancemgr.keikoproj.io/custom-networking-host-pods|InstanceGroup|"2"|setting this annotation increases the number of max pods on nodes with custom networking, due to the fact that hostNetwork pods do not use an additional IP address |


Test patch:
diff --git a/controllers/provisioners/eks/helpers_test.go b/controllers/provisioners/eks/helpers_test.go
index 5f16ed0..ec2d826 100644
--- a/controllers/provisioners/eks/helpers_test.go
+++ b/controllers/provisioners/eks/helpers_test.go
@@ -314,6 +314,16 @@ func TestCustomNetworkingMaxPods(t *testing.T) {
 			bootstrapOptions: nil,
 			expectedMaxPods:  "--max-pods=20",
 		},
+		{
+			annotations: map[string]string{
+				ClusterAutoscalerEnabledAnnotation: "true",
+				CustomNetworkingPrefixAssignmentEnabledAnnotation: "true",
+				CustomNetworkingHostPodsAnnotation: "2",
+				CustomNetworkingEnabledAnnotation:  "true",
+			},
+			bootstrapOptions: nil,
+			expectedMaxPods:  "--max-pods=290",
+		},
 		{
 			annotations: map[string]string{
 				ClusterAutoscalerEnabledAnnotation: "true",