Repository: kubernetes-sigs/cluster-api-provider-aws
Pull request #4667 (2023-11-29 03:36:25)

:bug: cleanup: eliminate log spam when using S3 secrets
/kind cleanup

Currently, the S3 object store generates a large amount of log spam during reconciliation. Objects are stated to be deleting regardless of objects already being deleted.

Example:
```
I1129 02:23:28.008345       1 logger.go:67] "Deleting unneeded entry from AWS S3" secretPrefix=""
I1129 02:23:28.008394       1 logger.go:67] "Deleting object" controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="clusters-system/us-west-2-production-worker-us-west-2d-infra-vln7r-xzcsk" namespace="clusters-system" name="us-west-2-production-worker-us-west-2d-infra-vln7r-xzcsk" reconcileID="c10cc649-57ce-40f4-99bb-17d06590c2cf" machine="clusters-system/us-west-2-production-worker-us-west-2d-5lgbh-6cbb9999d5xlnxtcvd" cluster="clusters-system/us-west-2-production" bucket_name="cluster-api-provider-aws-gremlin-us-west-2-production" key="node/us-west-2-production-worker-us-west-2d-infra-vln7r-xzcsk"
I1129 02:24:05.691604       1 awsmanagedcluster_controller.go:114] "Successfully reconciled AWSManagedCluster" controller="awsmanagedcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSManagedCluster" AWSManagedCluster="clusters-system/us-east-1-production-testing" namespace="clusters-system" name="us-east-1-production-testing" reconcileID="8cbaa551-85ef-4feb-9fac-533577193662" cluster="us-east-1-production-testing" controlPlane="us-east-1-production-testing-control-plane"
```

- [x] squashed commits
- [x] includes documentation (N/A)
- [x] includes [emojis](https://github.com/kubernetes-sigs/kubebuilder-release-tools?tab=readme-ov-file#kubebuilder-project-versioning)
- [x] adds unit tests (N/A
- [x] adds or updates e2e tests

**Release note**:
```release-note
If you're using S3 buckets for bootstrap data, the following permissions are required on the controller to automatically detect whether an object is still needing to be deleted: `s3:GetObject`. The following permissions are optional: `s3:ListBucket`.
```


k8s-ci-robot: Hi @thefirstofthe300. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcluster-api-provider-aws).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
**Once this PR has been reviewed and has the lgtm label**, please assign [ankitasw](https://github.com/ankitasw) for approval. For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcluster-api-provider-aws).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["ankitasw"]} -->
vincepri: /ok-to-test
vincepri: /test ?
k8s-ci-robot: @vincepri: The following commands are available to trigger required jobs:
* `/test pull-cluster-api-provider-aws-build`
* `/test pull-cluster-api-provider-aws-test`
* `/test pull-cluster-api-provider-aws-verify`

The following commands are available to trigger optional jobs:
* `/test pull-cluster-api-provider-aws-apidiff-main`
* `/test pull-cluster-api-provider-aws-e2e`
* `/test pull-cluster-api-provider-aws-e2e-blocking`
* `/test pull-cluster-api-provider-aws-e2e-clusterclass`
* `/test pull-cluster-api-provider-aws-e2e-conformance`
* `/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts`
* `/test pull-cluster-api-provider-aws-e2e-eks`
* `/test pull-cluster-api-provider-aws-e2e-eks-gc`
* `/test pull-cluster-api-provider-aws-e2e-eks-testing`

Use `/test all` to run the following jobs that were automatically triggered:
* `pull-cluster-api-provider-aws-apidiff-main`
* `pull-cluster-api-provider-aws-build`
* `pull-cluster-api-provider-aws-test`
* `pull-cluster-api-provider-aws-verify`



<details>

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4667#issuecomment-1832733235):

>/test ?


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
vincepri: /test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-clusterclass
/test pull-cluster-api-provider-aws-e2e-conformance
/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
/test pull-cluster-api-provider-aws-e2e-eks
/test pull-cluster-api-provider-aws-e2e-eks-gc
/test pull-cluster-api-provider-aws-e2e-eks-testing
k8s-ci-robot: @thefirstofthe300: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cluster-api-provider-aws-verify | abb23b7b2c4d77d2b05cc4988517dda8a2f354dc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/4667/pull-cluster-api-provider-aws-verify/1744484249827807232) | true | `/test pull-cluster-api-provider-aws-verify`
pull-cluster-api-provider-aws-test | abb23b7b2c4d77d2b05cc4988517dda8a2f354dc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/4667/pull-cluster-api-provider-aws-test/1744484249660035072) | true | `/test pull-cluster-api-provider-aws-test`
pull-cluster-api-provider-aws-apidiff-main | abb23b7b2c4d77d2b05cc4988517dda8a2f354dc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/4667/pull-cluster-api-provider-aws-apidiff-main/1744484249710366720) | false | `/test pull-cluster-api-provider-aws-apidiff-main`
pull-cluster-api-provider-aws-build | abb23b7b2c4d77d2b05cc4988517dda8a2f354dc | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/4667/pull-cluster-api-provider-aws-build/1744484249760698368) | true | `/test pull-cluster-api-provider-aws-build`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api-provider-aws&pr=4667). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Athefirstofthe300). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
thefirstofthe300: /retest

Patch:
diff --git a/cmd/clusterawsadm/cloudformation/bootstrap/cluster_api_controller.go b/cmd/clusterawsadm/cloudformation/bootstrap/cluster_api_controller.go
index 3b44b5ec2..14f8d423b 100644
--- a/cmd/clusterawsadm/cloudformation/bootstrap/cluster_api_controller.go
+++ b/cmd/clusterawsadm/cloudformation/bootstrap/cluster_api_controller.go
@@ -284,6 +284,7 @@ func (t Template) ControllersPolicy() *iamv1.PolicyDocument {
 			Action: iamv1.Actions{
 				"s3:CreateBucket",
 				"s3:DeleteBucket",
+				"s3:GetObject",
 				"s3:PutObject",
 				"s3:DeleteObject",
 				"s3:PutBucketPolicy",
diff --git a/cmd/clusterawsadm/cloudformation/bootstrap/fixtures/with_s3_bucket.yaml b/cmd/clusterawsadm/cloudformation/bootstrap/fixtures/with_s3_bucket.yaml
index 9344f79aa..39bd20ef2 100644
--- a/cmd/clusterawsadm/cloudformation/bootstrap/fixtures/with_s3_bucket.yaml
+++ b/cmd/clusterawsadm/cloudformation/bootstrap/fixtures/with_s3_bucket.yaml
@@ -289,6 +289,7 @@ Resources:
         - Action:
           - s3:CreateBucket
           - s3:DeleteBucket
+          - s3:GetObject
           - s3:PutObject
           - s3:DeleteObject
           - s3:PutBucketPolicy
diff --git a/controllers/awsmachine_controller.go b/controllers/awsmachine_controller.go
index 007429352..bec9ccaf4 100644
--- a/controllers/awsmachine_controller.go
+++ b/controllers/awsmachine_controller.go
@@ -850,8 +850,6 @@ func (r *AWSMachineReconciler) deleteIgnitionBootstrapDataFromS3(machineScope *s
 		return nil
 	}
 
-	machineScope.Info("Deleting unneeded entry from AWS S3", "secretPrefix", machineScope.GetSecretPrefix())
-
 	if err := objectStoreSvc.Delete(machineScope); err != nil {
 		return errors.Wrap(err, "deleting bootstrap data object")
 	}
diff --git a/pkg/cloud/services/s3/s3.go b/pkg/cloud/services/s3/s3.go
index cde883d64..4bccb6307 100644
--- a/pkg/cloud/services/s3/s3.go
+++ b/pkg/cloud/services/s3/s3.go
@@ -173,27 +173,52 @@ func (s *Service) Delete(m *scope.MachineScope) error {
 	bucket := s.bucketName()
 	key := s.bootstrapDataKey(m)
 
-	s.scope.Info("Deleting object", "bucket_name", bucket, "key", key)
-
-	_, err := s.S3Client.DeleteObject(&s3.DeleteObjectInput{
+	_, err := s.S3Client.HeadObject(&s3.HeadObjectInput{
 		Bucket: aws.String(bucket),
 		Key:    aws.String(key),
 	})
-	if err == nil {
-		return nil
-	}
-
-	aerr, ok := err.(awserr.Error)
-	if !ok {
+	if err != nil {
+		if aerr, ok := err.(awserr.Error); ok {
+			switch aerr.Code() {
+			case "Forbidden":
+				// In the case that the IAM policy does not have sufficient
+				// permissions to get the object, we will attempt to delete it
+				// anyway for backwards compatibility reasons.
+				s.scope.Debug("Received 403 forbidden from S3 HeadObject call. If GetObject permission has been granted to the controller but not ListBucket, object is already deleted. Attempting deletion anyway in case GetObject permission hasn't been granted to the controller but DeleteObject has.", "bucket", bucket, "key", key)
+
+				_, err = s.S3Client.DeleteObject(&s3.DeleteObjectInput{
+					Bucket: aws.String(bucket),
+					Key:    aws.String(key),
+				})
+				if err != nil {
+					return errors.Wrap(err, "deleting S3 object")
+				}
+
+				s.scope.Debug("Delete object call succeeded despite missing GetObject permission", "bucket", bucket, "key", key)
+
+				return nil
+			case s3.ErrCodeNoSuchKey:
+				s.scope.Debug("Object already deleted", "bucket", bucket, "key", key)
+				return nil
+			case s3.ErrCodeNoSuchBucket:
+				s.scope.Debug("Bucket does not exist", "bucket", bucket)
+				return nil
+			default:
+				return errors.Wrap(aerr, "deleting S3 object")
+			}
+		}
+	}
+
+	s.scope.Info("Deleting S3 object", "bucket", bucket, "key", key)
+
+	_, err = s.S3Client.DeleteObject(&s3.DeleteObjectInput{
+		Bucket: aws.String(bucket),
+		Key:    aws.String(key),
+	})
+	if err != nil {
 		return errors.Wrap(err, "deleting S3 object")
 	}
 
-	switch aerr.Code() {
-	case s3.ErrCodeNoSuchBucket:
-	default:
-		return errors.Wrap(aerr, "deleting S3 object")
-	}
-
 	return nil
 }
 


Test patch:
diff --git a/controllers/awsmachine_controller_unit_test.go b/controllers/awsmachine_controller_unit_test.go
index 44e7ef305..c330f5190 100644
--- a/controllers/awsmachine_controller_unit_test.go
+++ b/controllers/awsmachine_controller_unit_test.go
@@ -1136,7 +1136,6 @@ func TestAWSMachineReconciler(t *testing.T) {
 
 				_, err := reconciler.reconcileDelete(ms, cs, cs, cs, cs)
 				g.Expect(err).To(BeNil())
-				g.Expect(buf.String()).To(ContainSubstring("Deleting unneeded entry from AWS S3"))
 			})
 		})
 
diff --git a/pkg/cloud/services/s3/s3_test.go b/pkg/cloud/services/s3/s3_test.go
index 76e43691e..0f4483522 100644
--- a/pkg/cloud/services/s3/s3_test.go
+++ b/pkg/cloud/services/s3/s3_test.go
@@ -631,6 +631,8 @@ func TestDeleteObject(t *testing.T) {
 			},
 		}
 
+		s3Mock.EXPECT().HeadObject(gomock.Any())
+
 		s3Mock.EXPECT().DeleteObject(gomock.Any()).Do(func(deleteObjectInput *s3svc.DeleteObjectInput) {
 			t.Run("use_configured_bucket_name_on_cluster_level", func(t *testing.T) {
 				t.Parallel()
@@ -672,7 +674,7 @@ func TestDeleteObject(t *testing.T) {
 			},
 		}
 
-		s3Mock.EXPECT().DeleteObject(gomock.Any()).Return(nil, awserr.New(s3svc.ErrCodeNoSuchBucket, "", nil)).Times(1)
+		s3Mock.EXPECT().HeadObject(gomock.Any()).Return(nil, awserr.New(s3svc.ErrCodeNoSuchBucket, "", nil))
 
 		if err := svc.Delete(machineScope); err != nil {
 			t.Fatalf("Unexpected error, got: %v", err)
@@ -696,6 +698,7 @@ func TestDeleteObject(t *testing.T) {
 				},
 			}
 
+			s3Mock.EXPECT().HeadObject(gomock.Any())
 			s3Mock.EXPECT().DeleteObject(gomock.Any()).Return(nil, errors.New("foo")).Times(1)
 
 			if err := svc.Delete(machineScope); err == nil {
@@ -747,6 +750,7 @@ func TestDeleteObject(t *testing.T) {
 			},
 		}
 
+		s3Mock.EXPECT().HeadObject(gomock.Any()).Times(2)
 		s3Mock.EXPECT().DeleteObject(gomock.Any()).Return(nil, nil).Times(2)
 
 		if err := svc.Delete(machineScope); err != nil {