Repository: kubernetes-sigs/kube-scheduler-simulator (go)

Issue:
metadata.annotations: Too long: must have at most 262144 bytes
Running into the following issue


`E0519 08:10:15.950903       1 storereflector.go:137] failed to update the pod with retry to record store: update pod:`
`    sigs.k8s.io/kube-scheduler-simulator/simulator/scheduler/storereflector.(*reflector).ResisterResultSavingToInformer.(*reflector).storeAllResultToPodFunc.func1.1`
`        /go/src/simulator/scheduler/storereflector/storereflector.go:132`
`  - Pod "job-807-nfsdr" is invalid: metadata.annotations: Too long: must have at most 262144 bytes`

I assume it is because i have quite a lot of nodes in my setup (around 800), and the annotations that the debuggable scheduler makes therefore become too long. 

Is there perhaps a way to temporarily disable the annotations?

Pull request description:
fix: trim the history annotation when the size is too big
<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind bug
<!--
Add related area tag(s):
/area web
/area simulator
/area scenario

Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:
Delete old annotations when annotation string exceeds 256KB

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #417 

#### Special notes for your reviewer:


/label tide/merge-method-squash


Patch:
diff --git a/simulator/scheduler/storereflector/storereflector.go b/simulator/scheduler/storereflector/storereflector.go
index 7f6a426d..ffa2dc7d 100644
--- a/simulator/scheduler/storereflector/storereflector.go
+++ b/simulator/scheduler/storereflector/storereflector.go
@@ -9,6 +9,7 @@ import (
 	"golang.org/x/xerrors"
 	corev1 "k8s.io/api/core/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	validation "k8s.io/apimachinery/pkg/api/validation"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	clientset "k8s.io/client-go/kubernetes"
 	"k8s.io/client-go/tools/cache"
@@ -157,11 +158,19 @@ func updateResultHistory(p *corev1.Pod, m map[string]string) error {
 
 	results = append(results, m)
 
-	r, err := json.Marshal(results)
-	if err != nil {
-		return xerrors.Errorf("encode all results: %w", err)
+	// If we exceed that annotation size limitation, we should drop entries from the oldest side
+	// of the history slice until the payload fits, since the newer histories are likely more important.
+	for ; len(results) != 0; results = results[1:] {
+		r, err := json.Marshal(results)
+		if err != nil {
+			return xerrors.Errorf("encode all results: %w", err)
+		}
+		if len(r) <= validation.TotalAnnotationSizeLimitB {
+			metav1.SetMetaDataAnnotation(&p.ObjectMeta, ResultsHistoryAnnotation, string(r))
+			return nil
+		}
 	}
-	metav1.SetMetaDataAnnotation(&p.ObjectMeta, ResultsHistoryAnnotation, string(r))
 
-	return nil
+	// Basically shouldn't happen unless a single history entry is bigger than TotalAnnotationSizeLimitB, which is very unlikely.
+	return xerrors.Errorf("result history still exceeds annotation limit even after removing several histories")
 }


Test patch:
diff --git a/simulator/scheduler/storereflector/storereflector_test.go b/simulator/scheduler/storereflector/storereflector_test.go
index 84c19af0..1af0300d 100644
--- a/simulator/scheduler/storereflector/storereflector_test.go
+++ b/simulator/scheduler/storereflector/storereflector_test.go
@@ -3,6 +3,7 @@ package storereflector
 import (
 	"context"
 	"fmt"
+	"strings"
 	"testing"
 
 	"github.com/google/go-cmp/cmp"
@@ -129,6 +130,27 @@ func Test_updateResultHistory(t *testing.T) {
 			},
 			wantErr: assert.NoError,
 		},
+		{
+			name: "success: trim oldest result history when exceeds annotation size limitation",
+			p: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						ResultsHistoryAnnotation: fmt.Sprintf(`[{"result":"%s"}]`, strings.Repeat("a", 200000)),
+					},
+				},
+			},
+			m: map[string]string{
+				"result": strings.Repeat("b", 200000),
+			},
+			wantPod: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						ResultsHistoryAnnotation: fmt.Sprintf(`[{"result":"%s"}]`, strings.Repeat("b", 200000)),
+					},
+				},
+			},
+			wantErr: assert.NoError,
+		},
 		{
 			name: "fail: Pod has broken value on annotation",
 			p: &corev1.Pod{
@@ -144,6 +166,27 @@ func Test_updateResultHistory(t *testing.T) {
 			},
 			wantErr: assert.Error,
 		},
+		{
+			name: "fail: single result history exceeds annotation size limitation",
+			p: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						ResultsHistoryAnnotation: "[]",
+					},
+				},
+			},
+			m: map[string]string{
+				"result": strings.Repeat("a", 270000),
+			},
+			wantPod: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						ResultsHistoryAnnotation: "[]",
+					},
+				},
+			},
+			wantErr: assert.Error,
+		},
 	}
 	for _, tt := range tests {
 		tt := tt