Repository: kubernetes-sigs/cluster-api-provider-aws

Issue:
Kubernetes API server rolling restarts experience client-side disruption
/kind bug

**What steps did you take and what happened:**
[A clear and concise description of what the bug is.]

An [automated test](https://github.com/openshift/origin/blob/6767b922e0d15c31c3fc349c659987ce1922d650/test/extended/apiserver/rollout.go#L36) that forces kube API server restarts observed that roughly 30 seconds after the kube api-server returns HTTP 500s from `/readyz` pre-shutdown, clients are getting `ECONNRESET` when trying to read responses.

**What did you expect to happen:**

Clients would not have their connections interrupted.

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]

The `target_health_state.unhealthy.draining_interval_seconds` attribute should probably be set to something higher than the default (which is 0) in order to make client interactions more resilient to kube-api-server restarts.

This draining interval is separate from the health check interval seconds we currently set - that one just tells AWS how often to invoke the health check.

See the [TargetGroupAttribute docs](https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_TargetGroupAttribute.html) for specifics.

This was observed during OpenShift testing, but is applicable generally to anyone restarting their KAS.

/area networking
/triage accepted

Hints from issue comments:
@nrb: Reported by @benluddy @benluddy: More from the apiserver perspective:

When an apiserver gets SIGTERM, it immediate begins returning HTTP 500 from /readyz, but continues to accept new connections and serve existing connections for a period of time controlled by `--shutdown-delay-duration`. The purpose of this delay is to give load balancers enough time to mark the instance unhealthy and stop directing new traffic there. After this delay, the apiserver closes established idle connections and allows active connections a period of time to complete any in-flight requests and be closed (https://pkg.go.dev/net/http#Server.Shutdown).

If the load balancer closes established connections the moment an apiserver is considered unhealthy, any clients with in-flight requests observe errors.

This connection termination behavior seems to be unique to NLB (https://docs.aws.amazon.com/elasticloadbalancing/latest/network/edit-target-group-attributes.html#unhealthy-target-connection-termination). I don't _think_ there is a use case for leaving it enabled when the targets are kube-apiservers, since it defeats the mechanism for graceful shutdown.@nrb: I think setting a default drain period for the NLBs backing a kube api-server makes sense; as Ben points out, not doing so leads to ungraceful shutdowns and client disruption.

Given that, should we update our default value? And should this option even be exposed for users to change via our CRDs? I think if we allow users to clear it, which is the current default, we're setting them up for failure. If we allow changing it, I think it should be some value above a minimum.

Curious what others think.

cc @richardcase @dlipovetsky @AndiDog 

Solution patch:
diff --git a/api/v1beta2/network_types.go b/api/v1beta2/network_types.go
index 9974cbddb..00a775a54 100644
--- a/api/v1beta2/network_types.go
+++ b/api/v1beta2/network_types.go
@@ -207,6 +207,14 @@ type TargetGroupAttribute string
 var (
 	// TargetGroupAttributeEnablePreserveClientIP defines the attribute key for enabling preserve client IP.
 	TargetGroupAttributeEnablePreserveClientIP = "preserve_client_ip.enabled"
+
+	// TargetGroupAttributeEnableConnectionTermination defines the attribute key for terminating
+	// established connections to unhealthy targets.
+	TargetGroupAttributeEnableConnectionTermination = "target_health_state.unhealthy.connection_termination.enabled"
+
+	// TargetGroupAttributeUnhealthyDrainingIntervalSeconds defines the attribute key for the
+	// unhealthy target connection draining interval.
+	TargetGroupAttributeUnhealthyDrainingIntervalSeconds = "target_health_state.unhealthy.draining_interval_seconds"
 )
 
 // LoadBalancerAttribute defines a set of attributes for a V2 load balancer.
diff --git a/config/crd/bases/controlplane.cluster.x-k8s.io_rosacontrolplanes.yaml b/config/crd/bases/controlplane.cluster.x-k8s.io_rosacontrolplanes.yaml
index e9043004d..9f68e97ea 100644
--- a/config/crd/bases/controlplane.cluster.x-k8s.io_rosacontrolplanes.yaml
+++ b/config/crd/bases/controlplane.cluster.x-k8s.io_rosacontrolplanes.yaml
@@ -87,6 +87,7 @@ spec:
                 description: OpenShift version channel group, default is stable.
                 enum:
                 - stable
+                - fast
                 - candidate
                 - nightly
                 type: string
diff --git a/controlplane/rosa/api/v1beta2/rosacontrolplane_types.go b/controlplane/rosa/api/v1beta2/rosacontrolplane_types.go
index b7bc7a570..95d06281f 100644
--- a/controlplane/rosa/api/v1beta2/rosacontrolplane_types.go
+++ b/controlplane/rosa/api/v1beta2/rosacontrolplane_types.go
@@ -59,6 +59,9 @@ const (
 	// Stable channel group is the default channel group for stable releases.
 	Stable ChannelGroupType = "stable"
 
+	// Fast channel group is for fast channel releases.
+	Fast ChannelGroupType = "fast"
+
 	// Candidate channel group is for testing candidate builds.
 	Candidate ChannelGroupType = "candidate"
 
@@ -107,7 +110,7 @@ type RosaControlPlaneSpec struct { //nolint: maligned
 
 	// OpenShift version channel group, default is stable.
 	//
-	// +kubebuilder:validation:Enum=stable;candidate;nightly
+	// +kubebuilder:validation:Enum=stable;fast;candidate;nightly
 	// +kubebuilder:default=stable
 	ChannelGroup ChannelGroupType `json:"channelGroup"`
 
diff --git a/docs/book/src/topics/rosa/upgrades.md b/docs/book/src/topics/rosa/upgrades.md
index f668399d3..8eabf706b 100644
--- a/docs/book/src/topics/rosa/upgrades.md
+++ b/docs/book/src/topics/rosa/upgrades.md
@@ -4,14 +4,14 @@
 
 Upgrading the OpenShift version of the control plane is supported by the provider. To perform an upgrade you need to update the `version` in the spec of the `ROSAControlPlane`. Once the version has changed the provider will handle the upgrade for you.
 
-Upgrading y-stream version ex; v4.16.x to v4.17.x required the version gate acknowledgement. By default the versionGate is set to WaitForAcknowledge in the `ROSAControlPlane` CR. When upgrading to y-stream version the versionGate should be set to Acknowledge or AlwaysAcknowledge. 
+Upgrading y-stream version ex; v4.16.x to v4.17.x required the version gate acknowledgement. By default the versionGate is set to WaitForAcknowledge in the `ROSAControlPlane` CR. When upgrading to y-stream version the versionGate should be set to Acknowledge or AlwaysAcknowledge.
 
 ##### Note:
 When the versionGate is set to 'Acknowledge', it will revert to 'WaitForAcknowledge' once the upgrade is successfully completed. However, if the versionGate is set to 'AlwaysAcknowledge', it will remain set to 'AlwaysAcknowledge' after the upgrade is successfully completed.
 
 The available upgrades versions for the `ROSAControlPlane` will be listed under `ROSAControlPlane.status.availableUpgrades`
 
-The version channel group `ROSAControlPlane.spec.channelGroup` default to stable. However, it can be set to candidate or nightly. Changing the version channel group will change the `ROSAControlPlane.status.availableUpgrades` accordingly. 
+The version channel group `ROSAControlPlane.spec.channelGroup` default to stable. However, it can be set to fast, candidate, or nightly. Changing the version channel group will change the `ROSAControlPlane.status.availableUpgrades` accordingly. Note that the use of channel groups other than stable may require additional permissions.
 
 The Upgrade state can be checked in the conditions under `ROSAControlPlane.status`.
 
diff --git a/go.mod b/go.mod
index 23abc3af9..188afcece 100644
--- a/go.mod
+++ b/go.mod
@@ -139,7 +139,7 @@ require (
 	github.com/godbus/dbus/v5 v5.1.0 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
-	github.com/golang/glog v1.2.4 // indirect
+	github.com/golang/glog v1.2.5 // indirect
 	github.com/golang/protobuf v1.5.4 // indirect
 	github.com/google/btree v1.1.3 // indirect
 	github.com/google/cel-go v0.22.0 // indirect
diff --git a/go.sum b/go.sum
index 59623666d..f140c2129 100644
--- a/go.sum
+++ b/go.sum
@@ -236,8 +236,8 @@ github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69
 github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI=
 github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
-github.com/golang/glog v1.2.4 h1:CNNw5U8lSiiBk7druxtSHHTsRWcxKoac6kZKm2peBBc=
-github.com/golang/glog v1.2.4/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w=
+github.com/golang/glog v1.2.5 h1:DrW6hGnjIhtvhOIiAKT6Psh/Kd/ldepEa81DKeiRJ5I=
+github.com/golang/glog v1.2.5/go.mod h1:6AhwSGph0fcJtXVM/PEHPqZlFeoLxhs7/t5UDAwmO+w=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.6.0 h1:ErTB+efbowRARo13NNdxyJji2egdxLGQhRaY+DUumQc=
 github.com/golang/mock v1.6.0/go.mod h1:p6yTPP+5HYm5mzsMV8JkE6ZKdX+/wYM6Hr+LicevLPs=
diff --git a/pkg/cloud/services/elb/loadbalancer.go b/pkg/cloud/services/elb/loadbalancer.go
index 25aefa9b4..6887692dd 100644
--- a/pkg/cloud/services/elb/loadbalancer.go
+++ b/pkg/cloud/services/elb/loadbalancer.go
@@ -1687,16 +1687,32 @@ func (s *Service) reconcileTargetGroupsAndListeners(ctx context.Context, lbARN s
 			}
 			createdTargetGroups = append(createdTargetGroups, group)
 
+			targetGroupAttributeInput := &elbv2.ModifyTargetGroupAttributesInput{TargetGroupArn: group.TargetGroupArn}
+
+			if lbSpec.LoadBalancerType == infrav1.LoadBalancerTypeNLB {
+				targetGroupAttributeInput.Attributes = append(targetGroupAttributeInput.Attributes,
+					elbv2types.TargetGroupAttribute{
+						Key:   aws.String(infrav1.TargetGroupAttributeEnableConnectionTermination),
+						Value: aws.String("false"),
+					},
+					elbv2types.TargetGroupAttribute{
+						Key:   aws.String(infrav1.TargetGroupAttributeUnhealthyDrainingIntervalSeconds),
+						Value: aws.String("300"),
+					},
+				)
+			}
+
 			if !lbSpec.PreserveClientIP {
-				targetGroupAttributeInput := &elbv2.ModifyTargetGroupAttributesInput{
-					TargetGroupArn: group.TargetGroupArn,
-					Attributes: []elbv2types.TargetGroupAttribute{
-						{
-							Key:   aws.String(infrav1.TargetGroupAttributeEnablePreserveClientIP),
-							Value: aws.String("false"),
-						},
+				targetGroupAttributeInput.Attributes = append(targetGroupAttributeInput.Attributes,
+					elbv2types.TargetGroupAttribute{
+						Key:   aws.String(infrav1.TargetGroupAttributeEnablePreserveClientIP),
+						Value: aws.String("false"),
 					},
-				}
+				)
+			}
+
+			if len(targetGroupAttributeInput.Attributes) > 0 {
+				s.scope.Debug("configuring target group attributes", "attributes", targetGroupAttributeInput)
 				if _, err := s.ELBV2Client.ModifyTargetGroupAttributes(ctx, targetGroupAttributeInput); err != nil {
 					return nil, nil, errors.Wrapf(err, "failed to modify target group attribute")
 				}
diff --git a/pkg/cloud/services/ssm/mock_ssmiface/doc.go b/pkg/cloud/services/ssm/mock_ssmiface/doc.go
index 952d2ce8d..a80ac5bb7 100644
--- a/pkg/cloud/services/ssm/mock_ssmiface/doc.go
+++ b/pkg/cloud/services/ssm/mock_ssmiface/doc.go
@@ -14,9 +14,9 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-// Package mock_ssmiface provides a mock implementation for the IAMAPI interface.
+// Package mock_ssmiface provides a mock implementation for the SSMAPI interface.
 // Run go generate to regenerate this mock.
 //
-//go:generate ../../../../../hack/tools/bin/mockgen -destination ssmiface_mock.go -package mock_ssmiface sigs.k8s.io/cluster-api-provider-aws/v2/pkg/cloud/services/iamauth IAMAPI
-//go:generate /usr/bin/env bash -c "cat ../../../../../hack/boilerplate/boilerplate.generatego.txt mock_ssmiface.go > _ssmiface_mock.go && mv _ssmiface_mock.go ssmiface_mock.go"
+//go:generate ../../../../../hack/tools/bin/mockgen -destination mock_ssmiface.go -package mock_ssmiface sigs.k8s.io/cluster-api-provider-aws/v2/pkg/cloud/services/ssm SSMAPI
+//go:generate /usr/bin/env bash -c "cat ../../../../../hack/boilerplate/boilerplate.generatego.txt mock_ssmiface.go > _mock_ssmiface.go && mv _mock_ssmiface.go mock_ssmiface.go"
 package mock_ssmiface //nolint:stylecheck
diff --git a/pkg/cloud/services/ssm/mock_ssmiface/mock_ssmiface.go b/pkg/cloud/services/ssm/mock_ssmiface/mock_ssmiface.go
index 292f80408..dcc8202f4 100644
--- a/pkg/cloud/services/ssm/mock_ssmiface/mock_ssmiface.go
+++ b/pkg/cloud/services/ssm/mock_ssmiface/mock_ssmiface.go
@@ -1,5 +1,21 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+	http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
 // Code generated by MockGen. DO NOT EDIT.
-// Source: pkg/cloud/services/ssm/service.go
+// Source: sigs.k8s.io/cluster-api-provider-aws/v2/pkg/cloud/services/ssm (interfaces: SSMAPI)
 
 // Package mock_ssmiface is a generated GoMock package.
 package mock_ssmiface
@@ -36,10 +52,10 @@ func (m *MockSSMAPI) EXPECT() *MockSSMAPIMockRecorder {
 }
 
 // DeleteParameter mocks base method.
-func (m *MockSSMAPI) DeleteParameter(ctx context.Context, input *ssm.DeleteParameterInput, optFns ...func(*ssm.Options)) (*ssm.DeleteParameterOutput, error) {
+func (m *MockSSMAPI) DeleteParameter(arg0 context.Context, arg1 *ssm.DeleteParameterInput, arg2 ...func(*ssm.Options)) (*ssm.DeleteParameterOutput, error) {
 	m.ctrl.T.Helper()
-	varargs := []interface{}{ctx, input}
-	for _, a := range optFns {
+	varargs := []interface{}{arg0, arg1}
+	for _, a := range arg2 {
 		varargs = append(varargs, a)
 	}
 	ret := m.ctrl.Call(m, "DeleteParameter", varargs...)
@@ -49,17 +65,17 @@ func (m *MockSSMAPI) DeleteParameter(ctx context.Context, input *ssm.DeleteParam
 }
 
 // DeleteParameter indicates an expected call of DeleteParameter.
-func (mr *MockSSMAPIMockRecorder) DeleteParameter(ctx, input interface{}, optFns ...interface{}) *gomock.Call {
+func (mr *MockSSMAPIMockRecorder) DeleteParameter(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	varargs := append([]interface{}{ctx, input}, optFns...)
+	varargs := append([]interface{}{arg0, arg1}, arg2...)
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DeleteParameter", reflect.TypeOf((*MockSSMAPI)(nil).DeleteParameter), varargs...)
 }
 
 // GetParameter mocks base method.
-func (m *MockSSMAPI) GetParameter(ctx context.Context, input *ssm.GetParameterInput, optFns ...func(*ssm.Options)) (*ssm.GetParameterOutput, error) {
+func (m *MockSSMAPI) GetParameter(arg0 context.Context, arg1 *ssm.GetParameterInput, arg2 ...func(*ssm.Options)) (*ssm.GetParameterOutput, error) {
 	m.ctrl.T.Helper()
-	varargs := []interface{}{ctx, input}
-	for _, a := range optFns {
+	varargs := []interface{}{arg0, arg1}
+	for _, a := range arg2 {
 		varargs = append(varargs, a)
 	}
 	ret := m.ctrl.Call(m, "GetParameter", varargs...)
@@ -69,17 +85,17 @@ func (m *MockSSMAPI) GetParameter(ctx context.Context, input *ssm.GetParameterIn
 }
 
 // GetParameter indicates an expected call of GetParameter.
-func (mr *MockSSMAPIMockRecorder) GetParameter(ctx, input interface{}, optFns ...interface{}) *gomock.Call {
+func (mr *MockSSMAPIMockRecorder) GetParameter(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	varargs := append([]interface{}{ctx, input}, optFns...)
+	varargs := append([]interface{}{arg0, arg1}, arg2...)
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetParameter", reflect.TypeOf((*MockSSMAPI)(nil).GetParameter), varargs...)
 }
 
 // PutParameter mocks base method.
-func (m *MockSSMAPI) PutParameter(ctx context.Context, input *ssm.PutParameterInput, optFns ...func(*ssm.Options)) (*ssm.PutParameterOutput, error) {
+func (m *MockSSMAPI) PutParameter(arg0 context.Context, arg1 *ssm.PutParameterInput, arg2 ...func(*ssm.Options)) (*ssm.PutParameterOutput, error) {
 	m.ctrl.T.Helper()
-	varargs := []interface{}{ctx, input}
-	for _, a := range optFns {
+	varargs := []interface{}{arg0, arg1}
+	for _, a := range arg2 {
 		varargs = append(varargs, a)
 	}
 	ret := m.ctrl.Call(m, "PutParameter", varargs...)
@@ -89,8 +105,8 @@ func (m *MockSSMAPI) PutParameter(ctx context.Context, input *ssm.PutParameterIn
 }
 
 // PutParameter indicates an expected call of PutParameter.
-func (mr *MockSSMAPIMockRecorder) PutParameter(ctx, input interface{}, optFns ...interface{}) *gomock.Call {
+func (mr *MockSSMAPIMockRecorder) PutParameter(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	varargs := append([]interface{}{ctx, input}, optFns...)
+	varargs := append([]interface{}{arg0, arg1}, arg2...)
 	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "PutParameter", reflect.TypeOf((*MockSSMAPI)(nil).PutParameter), varargs...)
 }



Test patch:
diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go
index d68669c40..05f103cfb 100644
--- a/controllers/helpers_test.go
+++ b/controllers/helpers_test.go
@@ -393,6 +393,14 @@ func mockedModifyTargetGroupAttributes(t *testing.T, m *mocks.MockELBV2APIMockRe
 	m.ModifyTargetGroupAttributes(gomock.Any(), gomock.Eq(&elbv2.ModifyTargetGroupAttributesInput{
 		TargetGroupArn: tgArn,
 		Attributes: []elbv2types.TargetGroupAttribute{
+			{
+				Key:   aws.String(infrav1.TargetGroupAttributeEnableConnectionTermination),
+				Value: aws.String("false"),
+			},
+			{
+				Key:   aws.String(infrav1.TargetGroupAttributeUnhealthyDrainingIntervalSeconds),
+				Value: aws.String("300"),
+			},
 			{
 				Key:   aws.String(infrav1.TargetGroupAttributeEnablePreserveClientIP),
 				Value: aws.String("false"),
diff --git a/pkg/cloud/services/elb/loadbalancer_test.go b/pkg/cloud/services/elb/loadbalancer_test.go
index 805167b3b..5bf8baae0 100644
--- a/pkg/cloud/services/elb/loadbalancer_test.go
+++ b/pkg/cloud/services/elb/loadbalancer_test.go
@@ -1543,6 +1543,14 @@ func TestReconcileTargetGroupsAndListeners(t *testing.T) {
 				m.ModifyTargetGroupAttributes(gomock.Any(), gomock.Eq(&elbv2.ModifyTargetGroupAttributesInput{
 					TargetGroupArn: aws.String(tgArn),
 					Attributes: []elbv2types.TargetGroupAttribute{
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeEnableConnectionTermination),
+							Value: aws.String("false"),
+						},
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeUnhealthyDrainingIntervalSeconds),
+							Value: aws.String("300"),
+						},
 						{
 							Key:   aws.String(infrav1.TargetGroupAttributeEnablePreserveClientIP),
 							Value: aws.String("false"),
@@ -1661,6 +1669,14 @@ func TestReconcileTargetGroupsAndListeners(t *testing.T) {
 				m.ModifyTargetGroupAttributes(gomock.Any(), &elbv2.ModifyTargetGroupAttributesInput{
 					TargetGroupArn: aws.String(tgArn),
 					Attributes: []elbv2types.TargetGroupAttribute{
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeEnableConnectionTermination),
+							Value: aws.String("false"),
+						},
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeUnhealthyDrainingIntervalSeconds),
+							Value: aws.String("300"),
+						},
 						{
 							Key:   aws.String(infrav1.TargetGroupAttributeEnablePreserveClientIP),
 							Value: aws.String("false"),
@@ -1767,6 +1783,14 @@ func TestReconcileTargetGroupsAndListeners(t *testing.T) {
 				m.ModifyTargetGroupAttributes(gomock.Any(), &elbv2.ModifyTargetGroupAttributesInput{
 					TargetGroupArn: aws.String(tgArn),
 					Attributes: []elbv2types.TargetGroupAttribute{
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeEnableConnectionTermination),
+							Value: aws.String("false"),
+						},
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeUnhealthyDrainingIntervalSeconds),
+							Value: aws.String("300"),
+						},
 						{
 							Key:   aws.String(infrav1.TargetGroupAttributeEnablePreserveClientIP),
 							Value: aws.String("false"),
@@ -1860,6 +1884,19 @@ func TestReconcileTargetGroupsAndListeners(t *testing.T) {
 						},
 					},
 				}, nil)
+				m.ModifyTargetGroupAttributes(gomock.Any(), gomock.Eq(&elbv2.ModifyTargetGroupAttributesInput{
+					TargetGroupArn: aws.String(tgArn),
+					Attributes: []elbv2types.TargetGroupAttribute{
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeEnableConnectionTermination),
+							Value: aws.String("false"),
+						},
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeUnhealthyDrainingIntervalSeconds),
+							Value: aws.String("300"),
+						},
+					},
+				})).Return(nil, nil)
 				m.DescribeListeners(gomock.Any(), &elbv2.DescribeListenersInput{
 					LoadBalancerArn: aws.String(elbArn),
 				}).Return(&elbv2.DescribeListenersOutput{
@@ -2000,6 +2037,14 @@ func TestReconcileTargetGroupsAndListeners(t *testing.T) {
 				m.ModifyTargetGroupAttributes(gomock.Any(), &elbv2.ModifyTargetGroupAttributesInput{
 					TargetGroupArn: aws.String(tgArn),
 					Attributes: []elbv2types.TargetGroupAttribute{
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeEnableConnectionTermination),
+							Value: aws.String("false"),
+						},
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeUnhealthyDrainingIntervalSeconds),
+							Value: aws.String("300"),
+						},
 						{
 							Key:   aws.String(infrav1.TargetGroupAttributeEnablePreserveClientIP),
 							Value: aws.String("false"),
@@ -2115,6 +2160,14 @@ func TestReconcileTargetGroupsAndListeners(t *testing.T) {
 				m.ModifyTargetGroupAttributes(gomock.Any(), &elbv2.ModifyTargetGroupAttributesInput{
 					TargetGroupArn: aws.String(tgArn),
 					Attributes: []elbv2types.TargetGroupAttribute{
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeEnableConnectionTermination),
+							Value: aws.String("false"),
+						},
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeUnhealthyDrainingIntervalSeconds),
+							Value: aws.String("300"),
+						},
 						{
 							Key:   aws.String(infrav1.TargetGroupAttributeEnablePreserveClientIP),
 							Value: aws.String("false"),
@@ -2403,6 +2456,14 @@ func TestReconcileV2LB(t *testing.T) {
 				m.ModifyTargetGroupAttributes(gomock.Any(), gomock.Eq(&elbv2.ModifyTargetGroupAttributesInput{
 					TargetGroupArn: aws.String(tgArn),
 					Attributes: []elbv2types.TargetGroupAttribute{
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeEnableConnectionTermination),
+							Value: aws.String("false"),
+						},
+						{
+							Key:   aws.String(infrav1.TargetGroupAttributeUnhealthyDrainingIntervalSeconds),
+							Value: aws.String("300"),
+						},
 						{
 							Key:   aws.String(infrav1.TargetGroupAttributeEnablePreserveClientIP),
 							Value: aws.String("false"),