Repository: kserve/kserve
Pull request #4325: Feat: Allow inference service metadata injection at agent sidecar level for payload logging
Author: login: tylerhyang
Labels: name: lgtm
name: approved
Created: 2025-03-21T00:34:14Z | Merged: 2025-04-18T22:18:08Z

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://www.kubeflow.org/docs/about/contributing/ and developer guide https://github.com/kserve/kserve/blob/master/docs/DEVELOPER_GUIDE.md
2. Before raising a PR, please run `make precommit` to check the code style.
3. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
4. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

**What this PR does / why we need it**:
The InferenceService logger spec is expanded to include an additional section call "metadata-annotations" which specifies annotations to equip to the logging cloudEvent. Currently, fetching metadata from annotations requires making additional API calls to the inferenceService. This could be streamlined if the inferenceService passes the specified annotations to the agent sidecar at start up time to be stored in memory for faster payload enrichments

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #4292 

**Type of changes**
Please delete options that are not relevant.

- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update

**Feature/Issue validation/testing**:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

- [x] Test 1: Equip InferenceService with metadata-annotations fields in the spec and expect it to be present in the cloudEvent emitted

- [x] Test 2: Equip Inference Service with metadata-annotations fields in spec with annotations that are not present on the inferenceService.  They should be omitted in the cloudEvent
- [x] Test 3: When no metadata annotations are specified, the inference logger does not break

- Logs

**Special notes for your reviewer**:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

No image changes are made

**Checklist**:

- [x] Have you added unit/e2e tests that prove your fix is effective or that this feature works?
- [x] Has code been commented, particularly in hard-to-understand areas?
- [x] Have you made corresponding changes to the documentation?

**Release note**:
<!--  Write your release note:
1. Enter your extended release note in the below block. If the PR requires additional action from users switching to the new release, include the string "action required".
3. If no release note is required, just write "NONE".
-->
```release-note
Allow inference service annotation injection for payload logging
```

**Re-running failed tests**

- `/rerun-all` - rerun all failed workflows.
- `/rerun-workflow <workflow name>` - rerun a specific failed workflow. Only one workflow name can be specified. Multiple /rerun-workflow commands are allowed per comment.

Linked issue #4292: Allow inference service metadata injection at agent sidecar level for payload logging
/kind feature

**Describe the solution you'd like**
The InferenceService logger spec is expanded to include an additional section call "metadata-annotations" which specifies annotations to equip to the logging cloudEvent.  Currently, fetching metadata from annotations requires making additional API calls to the inferenceService.  This could be streamlined if the inferenceService passes the specified annotations to the agent sidecar at start up time to be stored in memory for faster payload enrichment



**Links to the design documents:**
[annotation injection design doc](https://docs.google.com/document/d/1TfCCO11-40mxAvi_OndaXO5tNKytOeH4t6BA6V-DzOI/edit?usp=sharing)

Diff:
diff --git a/charts/kserve-crd/templates/serving.kserve.io_inferenceservices.yaml b/charts/kserve-crd/templates/serving.kserve.io_inferenceservices.yaml
index d232e1a64cf..1bc65d09891 100644
--- a/charts/kserve-crd/templates/serving.kserve.io_inferenceservices.yaml
+++ b/charts/kserve-crd/templates/serving.kserve.io_inferenceservices.yaml
@@ -2788,6 +2788,10 @@ spec:
                     type: object
                   logger:
                     properties:
+                      metadataAnnotations:
+                        items:
+                          type: string
+                        type: array
                       metadataHeaders:
                         items:
                           type: string
@@ -7298,6 +7302,10 @@ spec:
                     type: object
                   logger:
                     properties:
+                      metadataAnnotations:
+                        items:
+                          type: string
+                        type: array
                       metadataHeaders:
                         items:
                           type: string
@@ -20233,6 +20241,10 @@ spec:
                     type: object
                   logger:
                     properties:
+                      metadataAnnotations:
+                        items:
+                          type: string
+                        type: array
                       metadataHeaders:
                         items:
                           type: string
diff --git a/cmd/agent/main.go b/cmd/agent/main.go
index 94c67c5525a..6705176f065 100644
--- a/cmd/agent/main.go
+++ b/cmd/agent/main.go
@@ -25,6 +25,7 @@ import (
 	"net/url"
 	"os"
 	"strconv"
+	"strings"
 	"time"
 
 	"github.com/go-logr/zapr"
@@ -58,15 +59,16 @@ var (
 	configDir    = flag.String("config-dir", "/mnt/configs", "directory for model config files")
 	modelDir     = flag.String("model-dir", "/mnt/models", "directory for model files")
 	// logger flags
-	logUrl           = flag.String("log-url", "", "The URL to send request/response logs to")
-	workers          = flag.Int("workers", 5, "Number of workers")
-	sourceUri        = flag.String("source-uri", "", "The source URI to use when publishing cloudevents")
-	logMode          = flag.String("log-mode", string(v1beta1.LogAll), "Whether to log 'request', 'response' or 'all'")
-	inferenceService = flag.String("inference-service", "", "The InferenceService name to add as header to log events")
-	namespace        = flag.String("namespace", "", "The namespace to add as header to log events")
-	endpoint         = flag.String("endpoint", "", "The endpoint name to add as header to log events")
-	component        = flag.String("component", "", "The component name (predictor, explainer, transformer) to add as header to log events")
-	metadataHeaders  = flag.StringSlice("metadata-headers", nil, "Allow list of headers that will be passed down as metadata")
+	logUrl              = flag.String("log-url", "", "The URL to send request/response logs to")
+	workers             = flag.Int("workers", 5, "Number of workers")
+	sourceUri           = flag.String("source-uri", "", "The source URI to use when publishing cloudevents")
+	logMode             = flag.String("log-mode", string(v1beta1.LogAll), "Whether to log 'request', 'response' or 'all'")
+	inferenceService    = flag.String("inference-service", "", "The InferenceService name to add as header to log events")
+	namespace           = flag.String("namespace", "", "The namespace to add as header to log events")
+	endpoint            = flag.String("endpoint", "", "The endpoint name to add as header to log events")
+	component           = flag.String("component", "", "The component name (predictor, explainer, transformer) to add as header to log events")
+	metadataHeaders     = flag.StringSlice("metadata-headers", nil, "Allow list of headers that will be passed down as metadata")
+	metadataAnnotations = flag.StringSlice("metadata-annotations", nil, "Allow list of metadata annotation to be passed with payload logging")
 	// batcher flags
 	enableBatcher = flag.Bool("enable-batcher", false, "Enable request batcher")
 	maxBatchSize  = flag.String("max-batchsize", "32", "Max Batch Size")
@@ -116,6 +118,7 @@ type loggerArgs struct {
 	endpoint         string
 	component        string
 	metadataHeaders  []string
+	annotations      map[string]string
 	certName         string
 	tlsSkipVerify    bool
 }
@@ -286,6 +289,17 @@ func startLogger(workers int, logger *zap.SugaredLogger) *loggerArgs {
 		os.Exit(-1)
 	}
 
+	var annotationKVPair map[string]string = map[string]string{}
+	for _, annotations := range *metadataAnnotations {
+		k, v, found := strings.Cut(annotations, "=")
+		if found {
+			annotationKVPair[k] = v
+		} else {
+			logger.Errorf("annotation does not adhere to desired format got key: %s value: %s", k, v)
+			os.Exit(-1)
+		}
+	}
+
 	logger.Info("Starting the log dispatcher")
 	kfslogger.StartDispatcher(workers, logger)
 	return &loggerArgs{
@@ -297,6 +311,7 @@ func startLogger(workers int, logger *zap.SugaredLogger) *loggerArgs {
 		namespace:        *namespace,
 		component:        *component,
 		metadataHeaders:  *metadataHeaders,
+		annotations:      annotationKVPair,
 		certName:         *CaCertFile,
 		tlsSkipVerify:    *TlsSkipVerify,
 	}
@@ -359,7 +374,7 @@ func buildServer(port string, userPort int, loggerArgs *loggerArgs, batcherArgs
 	if loggerArgs != nil {
 		composedHandler = kfslogger.New(loggerArgs.logUrl, loggerArgs.sourceUrl, loggerArgs.loggerType,
 			loggerArgs.inferenceService, loggerArgs.namespace, loggerArgs.endpoint, loggerArgs.component, composedHandler,
-			loggerArgs.metadataHeaders, loggerArgs.certName, loggerArgs.tlsSkipVerify)
+			loggerArgs.metadataHeaders, loggerArgs.certName, loggerArgs.annotations, loggerArgs.tlsSkipVerify)
 	}
 
 	composedHandler = queue.ForwardedShimHandler(composedHandler)
diff --git a/config/crd/full/serving.kserve.io_inferenceservices.yaml b/config/crd/full/serving.kserve.io_inferenceservices.yaml
index 6b598d15e9e..a5dc0023ee8 100644
--- a/config/crd/full/serving.kserve.io_inferenceservices.yaml
+++ b/config/crd/full/serving.kserve.io_inferenceservices.yaml
@@ -2788,6 +2788,10 @@ spec:
                       type: object
                     logger:
                       properties:
+                        metadataAnnotations:
+                          items:
+                            type: string
+                          type: array
                         metadataHeaders:
                           items:
                             type: string
@@ -7298,6 +7302,10 @@ spec:
                       type: object
                     logger:
                       properties:
+                        metadataAnnotations:
+                          items:
+                            type: string
+                          type: array
                         metadataHeaders:
                           items:
                             type: string
@@ -20233,6 +20241,10 @@ spec:
                       type: object
                     logger:
                       properties:
+                        metadataAnnotations:
+                          items:
+                            type: string
+                          type: array
                         metadataHeaders:
                           items:
                             type: string
diff --git a/hack/violation_exceptions.list b/hack/violation_exceptions.list
index d94b437df2a..d51dd5d9c1f 100644
--- a/hack/violation_exceptions.list
+++ b/hack/violation_exceptions.list
@@ -17,6 +17,7 @@ API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,InferenceServiceList,Items
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,InferenceServicesConfig,ServiceAnnotationDisallowedList
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,InferenceServicesConfig,ServiceLabelDisallowedList
+API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,LoggerSpec,MetadataAnnotations
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,LoggerSpec,MetadataHeaders
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,MultiNodeConfig,CustomGPUResourceTypeList
 API rule violation: list_type_missing,github.com/kserve/kserve/pkg/apis/serving/v1beta1,PodMetrics,MetricNames
diff --git a/pkg/apis/serving/v1beta1/inference_service.go b/pkg/apis/serving/v1beta1/inference_service.go
index d3a7b3537be..d548c121406 100644
--- a/pkg/apis/serving/v1beta1/inference_service.go
+++ b/pkg/apis/serving/v1beta1/inference_service.go
@@ -64,6 +64,9 @@ type LoggerSpec struct {
 	// Matched metadata HTTP headers for propagating to inference logger cloud events.
 	// +optional
 	MetadataHeaders []string `json:"metadataHeaders,omitempty"`
+	// Matched inference service annotations for propagating to inference logger cloud events.
+	// +optional
+	MetadataAnnotations []string `json:"metadataAnnotations,omitempty"`
 }
 
 // MetricsBackend enum
diff --git a/pkg/apis/serving/v1beta1/zz_generated.deepcopy.go b/pkg/apis/serving/v1beta1/zz_generated.deepcopy.go
index 0accb64150e..ed98c042852 100644
--- a/pkg/apis/serving/v1beta1/zz_generated.deepcopy.go
+++ b/pkg/apis/serving/v1beta1/zz_generated.deepcopy.go
@@ -542,6 +542,11 @@ func (in *LoggerSpec) DeepCopyInto(out *LoggerSpec) {
 		*out = make([]string, len(*in))
 		copy(*out, *in)
 	}
+	if in.MetadataAnnotations != nil {
+		in, out := &in.MetadataAnnotations, &out.MetadataAnnotations
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new LoggerSpec.
diff --git a/pkg/constants/constants.go b/pkg/constants/constants.go
index c82fb4483d9..85c2e890c6e 100644
--- a/pkg/constants/constants.go
+++ b/pkg/constants/constants.go
@@ -120,6 +120,7 @@ var (
 	LoggerSinkUrlInternalAnnotationKey               = InferenceServiceInternalAnnotationsPrefix + "/logger-sink-url"
 	LoggerModeInternalAnnotationKey                  = InferenceServiceInternalAnnotationsPrefix + "/logger-mode"
 	LoggerMetadataHeadersInternalAnnotationKey       = InferenceServiceInternalAnnotationsPrefix + "/logger-metadata-headers"
+	LoggerMetadataAnnotationsInternalAnnotationKey   = InferenceServiceInternalAnnotationsPrefix + "/logger-metadata-annotations"
 	BatcherInternalAnnotationKey                     = InferenceServiceInternalAnnotationsPrefix + "/batcher"
 	BatcherMaxBatchSizeInternalAnnotationKey         = InferenceServiceInternalAnnotationsPrefix + "/batcher-max-batchsize"
 	BatcherMaxLatencyInternalAnnotationKey           = InferenceServiceInternalAnnotationsPrefix + "/batcher-max-latency"
diff --git a/pkg/controller/v1beta1/inferenceservice/components/component.go b/pkg/controller/v1beta1/inferenceservice/components/component.go
index 7f987445697..a0f97da35d3 100644
--- a/pkg/controller/v1beta1/inferenceservice/components/component.go
+++ b/pkg/controller/v1beta1/inferenceservice/components/component.go
@@ -68,6 +68,9 @@ func addLoggerAnnotations(logger *v1beta1.LoggerSpec, annotations map[string]str
 		if logger.MetadataHeaders != nil {
 			annotations[constants.LoggerMetadataHeadersInternalAnnotationKey] = strings.Join(logger.MetadataHeaders, ",")
 		}
+		if logger.MetadataAnnotations != nil {
+			annotations[constants.LoggerMetadataAnnotationsInternalAnnotationKey] = strings.Join(logger.MetadataAnnotations, ",")
+		}
 	}
 }
 
diff --git a/pkg/logger/handler.go b/pkg/logger/handler.go
index 050bf0b5da5..0ee728d4491 100644
--- a/pkg/logger/handler.go
+++ b/pkg/logger/handler.go
@@ -83,13 +83,14 @@ type LoggerHandler struct {
 	endpoint         string
 	next             http.Handler
 	metadataHeaders  []string
+	annotations      map[string]string
 	certName         string
 	tlsSkipVerify    bool
 }
 
 func New(logUrl *url.URL, sourceUri *url.URL, logMode v1beta1.LoggerType,
 	inferenceService string, namespace string, endpoint string, component string, next http.Handler, metadataHeaders []string,
-	certName string, tlsSkipVerify bool,
+	certName string, annotations map[string]string, tlsSkipVerify bool,
 ) http.Handler {
 	logf.SetLogger(zap.New())
 	return &LoggerHandler{
@@ -102,6 +103,7 @@ func New(logUrl *url.URL, sourceUri *url.URL, logMode v1beta1.LoggerType,
 		component:        component,
 		endpoint:         endpoint,
 		next:             next,
+		annotations:      annotations,
 		metadataHeaders:  metadataHeaders,
 		certName:         certName,
 		tlsSkipVerify:    tlsSkipVerify,
@@ -150,6 +152,7 @@ func (eh *LoggerHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 			Namespace:        eh.namespace,
 			Endpoint:         eh.endpoint,
 			Component:        eh.component,
+			Annotations:      eh.annotations,
 			Metadata:         metadata,
 			CertName:         eh.certName,
 			TlsSkipVerify:    eh.tlsSkipVerify,
@@ -183,6 +186,7 @@ func (eh *LoggerHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 				InferenceService: eh.inferenceService,
 				Namespace:        eh.namespace,
 				Endpoint:         eh.endpoint,
+				Annotations:      eh.annotations,
 				Component:        eh.component,
 				CertName:         eh.certName,
 				TlsSkipVerify:    eh.tlsSkipVerify,
diff --git a/pkg/logger/handler_test.go b/pkg/logger/handler_test.go
index 91064299e73..d3179e471ee 100644
--- a/pkg/logger/handler_test.go
+++ b/pkg/logger/handler_test.go
@@ -79,7 +79,7 @@ func TestLogger(t *testing.T) {
 	StartDispatcher(5, logger)
 	httpProxy := httputil.NewSingleHostReverseProxy(targetUri)
 	oh := New(logSvcUrl, sourceUri, v1beta1.LogAll, "mymodel", "default", "default",
-		"default", httpProxy, nil, "", true)
+		"default", httpProxy, nil, "", nil, true)
 
 	oh.ServeHTTP(w, r)
 
@@ -151,7 +151,79 @@ func TestLoggerWithMetadata(t *testing.T) {
 	StartDispatcher(5, logger)
 	httpProxy := httputil.NewSingleHostReverseProxy(targetUri)
 	oh := New(logSvcUrl, sourceUri, v1beta1.LogAll, "mymodel", "default", "default",
-		"default", httpProxy, []string{"Foo", "Fizz"}, "", true)
+		"default", httpProxy, []string{"Foo", "Fizz"}, "", nil, true)
+
+	oh.ServeHTTP(w, r)
+
+	resp := w.Result()
+	defer resp.Body.Close()
+	b2, _ := io.ReadAll(resp.Body)
+	g.Expect(b2).To(gomega.Equal(predictorResponse))
+	// get logRequest
+	<-responseChan
+	// get logResponse
+	<-responseChan
+}
+
+func TestLoggerWithAnnotation(t *testing.T) {
+	g := gomega.NewGomegaWithT(t)
+
+	predictorRequest := []byte(`{"instances":[[0,0,0]]}`)
+	predictorResponse := []byte(`{"instances":[[4,5,6]]}`)
+
+	responseChan := make(chan string)
+	// Start a local HTTP server
+	logSvc := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
+		b, err := io.ReadAll(req.Body)
+		g.Expect(err).ToNot(gomega.HaveOccurred())
+		responseChan <- string(b)
+		g.Expect(b).To(gomega.Or(gomega.Equal(predictorRequest), gomega.Equal(predictorResponse)))
+		_, err = rw.Write([]byte(`ok`))
+		g.Expect(err).ToNot(gomega.HaveOccurred())
+
+		// If this is request, check the metadata
+		if req.Header["Ce-Type"][0] == "org.kubeflow.serving.inference.request" {
+			annotations := map[string]string{}
+
+			json.Unmarshal([]byte(req.Header["Ce-Annotations"][0]), &annotations)
+
+			g.Expect(annotations["Foo"]).To(gomega.Equal("Bar"))
+
+			g.Expect(annotations["Fizz"]).To(gomega.Equal("Buzz"))
+		}
+	}))
+	// Close the server when test finishes
+	defer logSvc.Close()
+
+	// Start a local HTTP server
+	predictor := httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
+		b, err := io.ReadAll(req.Body)
+		g.Expect(err).ToNot(gomega.HaveOccurred())
+		g.Expect(b).To(gomega.Or(gomega.Equal(predictorRequest), gomega.Equal(predictorResponse)))
+		_, err = rw.Write(predictorResponse)
+		g.Expect(err).ToNot(gomega.HaveOccurred())
+	}))
+	// Close the server when test finishes
+	defer predictor.Close()
+
+	reader := bytes.NewReader(predictorRequest)
+	r := httptest.NewRequest(http.MethodPost, "http://a", reader)
+	r.Header.Add("Foo", "bar")
+	r.Header.Add("Fizz", "buzz")
+	w := httptest.NewRecorder()
+	logger, _ := pkglogging.NewLogger("", "INFO")
+	logf.SetLogger(zap.New())
+	logSvcUrl, err := url.Parse(logSvc.URL)
+	g.Expect(err).ToNot(gomega.HaveOccurred())
+	sourceUri, err := url.Parse("http://localhost:9081/")
+	g.Expect(err).ToNot(gomega.HaveOccurred())
+	targetUri, err := url.Parse(predictor.URL)
+	g.Expect(err).ToNot(gomega.HaveOccurred())
+
+	StartDispatcher(5, logger)
+	httpProxy := httputil.NewSingleHostReverseProxy(targetUri)
+	oh := New(logSvcUrl, sourceUri, v1beta1.LogAll, "mymodel", "default", "default",
+		"default", httpProxy, nil, "", map[string]string{"Foo": "Bar", "Fizz": "Buzz"}, true)
 
 	oh.ServeHTTP(w, r)
 
@@ -196,7 +268,7 @@ func TestBadResponse(t *testing.T) {
 	StartDispatcher(1, logger)
 	httpProxy := httputil.NewSingleHostReverseProxy(targetUri)
 	oh := New(logSvcUrl, sourceUri, v1beta1.LogAll, "mymodel", "default", "default",
-		"default", httpProxy, nil, "", true)
+		"default", httpProxy, nil, "", nil, true)
 
 	oh.ServeHTTP(w, r)
 	g.Expect(w.Code).To(gomega.Equal(400))
diff --git a/pkg/logger/types.go b/pkg/logger/types.go
index 94ec450b249..b11aa2b8870 100644
--- a/pkg/logger/types.go
+++ b/pkg/logger/types.go
@@ -32,6 +32,7 @@ type LogRequest struct {
 	Component        string
 	Endpoint         string
 	Metadata         map[string][]string
+	Annotations      map[string]string
 	CertName         string
 	TlsSkipVerify    bool
 }
diff --git a/pkg/logger/worker.go b/pkg/logger/worker.go
index 7a98f43b22a..dfdfdbe5bd1 100644
--- a/pkg/logger/worker.go
+++ b/pkg/logger/worker.go
@@ -45,7 +45,8 @@ const (
 	ComponentAttr        = "component"
 	MetadataAttr         = "metadata"
 	// endpoint would be either default or canary
-	EndpointAttr = "endpoint"
+	EndpointAttr   = "endpoint"
+	AnnotationAttr = "annotations"
 
 	LoggerWorkerQueueSize = 100
 	CloudEventsIdHeader   = "Ce-Id"
@@ -133,6 +134,15 @@ func (w *Worker) sendCloudEvent(logReq LogRequest) error {
 	}
 	event.SetExtension(MetadataAttr, string(encodedMetadata))
 
+	if len(logReq.Annotations) > 0 {
+		bits, err := json.Marshal(logReq.Annotations)
+		if err != nil {
+			w.Log.Errorf("failed to marshal annotations: %w", err)
+		} else {
+			event.SetExtension(AnnotationAttr, string(bits))
+		}
+	}
+
 	event.SetSource(logReq.SourceUri.String())
 	if err := event.SetData(logReq.ContentType, *logReq.Bytes); err != nil {
 		return fmt.Errorf("while setting cloudevents data: %w", err)
diff --git a/pkg/openapi/openapi_generated.go b/pkg/openapi/openapi_generated.go
index 30c105a6c38..2a17304a12b 100644
--- a/pkg/openapi/openapi_generated.go
+++ b/pkg/openapi/openapi_generated.go
@@ -6325,6 +6325,21 @@ func schema_pkg_apis_serving_v1beta1_LoggerSpec(ref common.ReferenceCallback) co
 							},
 						},
 					},
+					"metadataAnnotations": {
+						SchemaProps: spec.SchemaProps{
+							Description: "Matched inference service annotations for propagating to inference logger cloud events.",
+							Type:        []string{"array"},
+							Items: &spec.SchemaOrArray{
+								Schema: &spec.Schema{
+									SchemaProps: spec.SchemaProps{
+										Default: "",
+										Type:    []string{"string"},
+										Format:  "",
+									},
+								},
+							},
+						},
+					},
 				},
 			},
 		},
diff --git a/pkg/openapi/swagger.json b/pkg/openapi/swagger.json
index 3683381fd3e..6c80b2cc315 100644
--- a/pkg/openapi/swagger.json
+++ b/pkg/openapi/swagger.json
@@ -3479,6 +3479,14 @@
       "description": "LoggerSpec specifies optional payload logging available for all components",
       "type": "object",
       "properties": {
+        "metadataAnnotations": {
+          "description": "Matched inference service annotations for propagating to inference logger cloud events.",
+          "type": "array",
+          "items": {
+            "type": "string",
+            "default": ""
+          }
+        },
         "metadataHeaders": {
           "description": "Matched metadata HTTP headers for propagating to inference logger cloud events.",
           "type": "array",
diff --git a/pkg/webhook/admission/pod/agent_injector.go b/pkg/webhook/admission/pod/agent_injector.go
index 7e822a8e475..29aaaffa8a7 100644
--- a/pkg/webhook/admission/pod/agent_injector.go
+++ b/pkg/webhook/admission/pod/agent_injector.go
@@ -34,17 +34,18 @@ import (
 )
 
 const (
-	LoggerConfigMapKeyName         = "logger"
-	LoggerArgumentLogUrl           = "--log-url"
-	LoggerArgumentSourceUri        = "--source-uri"
-	LoggerArgumentMode             = "--log-mode"
-	LoggerArgumentInferenceService = "--inference-service"
-	LoggerArgumentNamespace        = "--namespace"
-	LoggerArgumentEndpoint         = "--endpoint"
-	LoggerArgumentComponent        = "--component"
-	LoggerArgumentCaCertFile       = "--logger-ca-cert-file"
-	LoggerArgumentTlsSkipVerify    = "--logger-tls-skip-verify"
-	LoggerArgumentMetadataHeaders  = "--metadata-headers"
+	LoggerConfigMapKeyName            = "logger"
+	LoggerArgumentLogUrl              = "--log-url"
+	LoggerArgumentSourceUri           = "--source-uri"
+	LoggerArgumentMode                = "--log-mode"
+	LoggerArgumentInferenceService    = "--inference-service"
+	LoggerArgumentNamespace           = "--namespace"
+	LoggerArgumentEndpoint            = "--endpoint"
+	LoggerArgumentComponent           = "--component"
+	LoggerArgumentCaCertFile          = "--logger-ca-cert-file"
+	LoggerArgumentTlsSkipVerify       = "--logger-tls-skip-verify"
+	LoggerArgumentMetadataHeaders     = "--metadata-headers"
+	LoggerArgumentMetadataAnnotations = "--metadata-annotations"
 )
 
 type AgentConfig struct {
@@ -212,6 +213,20 @@ func (ag *AgentInjector) InjectAgent(pod *corev1.Pod) error {
 			loggerArgs = append(loggerArgs, LoggerArgumentMetadataHeaders)
 			loggerArgs = append(loggerArgs, logHeaderMetadata)
 		}
+		logMetadataAnnotations, ok := pod.ObjectMeta.Annotations[constants.LoggerMetadataAnnotationsInternalAnnotationKey]
+		if ok {
+			annotationKeys := strings.Split(logMetadataAnnotations, ",")
+			kvPairs := []string{}
+			for _, metadataAnnotation := range annotationKeys {
+				val, exists := pod.ObjectMeta.Annotations[metadataAnnotation]
+				if exists {
+					kvPairs = append(kvPairs, fmt.Sprintf("%s=%s", metadataAnnotation, val))
+				} else {
+					klog.Warningf("failed to find matching annotation %s on inference service", metadataAnnotation)
+				}
+			}
+			loggerArgs = append(loggerArgs, LoggerArgumentMetadataAnnotations, strings.Join(kvPairs, ","))
+		}
 		args = append(args, loggerArgs...)
 
 		// Add TLS cert name if specified. If not specified it will fall back to the arg's default.
diff --git a/pkg/webhook/admission/pod/agent_injector_test.go b/pkg/webhook/admission/pod/agent_injector_test.go
index 3061e591b4b..863948b225f 100644
--- a/pkg/webhook/admission/pod/agent_injector_test.go
+++ b/pkg/webhook/admission/pod/agent_injector_test.go
@@ -498,6 +498,138 @@ func TestAgentInjector(t *testing.T) {
 				},
 			},
 		},
+		"AddLoggerWithAnnotation": {
+			original: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "deployment",
+					Namespace: "default",
+					Annotations: map[string]string{
+						constants.LoggerInternalAnnotationKey:                    "true",
+						constants.LoggerSinkUrlInternalAnnotationKey:             "http://httpbin.org/",
+						constants.LoggerModeInternalAnnotationKey:                string(v1beta1.LogAll),
+						constants.LoggerMetadataAnnotationsInternalAnnotationKey: "Foo,Fizz",
+						"Foo":  "Bar",
+						"Fizz": "Buzz",
+					},
+					Labels: map[string]string{
+						"serving.kserve.io/inferenceservice": "sklearn",
+						constants.KServiceModelLabel:         "sklearn",
+						constants.KServiceEndpointLabel:      "default",
+						constants.KServiceComponentLabel:     "predictor",
+					},
+				},
+				Spec: corev1.PodSpec{
+					Containers: []corev1.Container{
+						{
+							Name: "sklearn",
+							ReadinessProbe: &corev1.Probe{
+								ProbeHandler: corev1.ProbeHandler{
+									TCPSocket: &corev1.TCPSocketAction{
+										Port: intstr.IntOrString{
+											IntVal: 8080,
+										},
+									},
+								},
+								InitialDelaySeconds: 0,
+								TimeoutSeconds:      1,
+								PeriodSeconds:       10,
+								SuccessThreshold:    1,
+								FailureThreshold:    3,
+							},
+						},
+						{
+							Name: "queue-proxy",
+							Env:  []corev1.EnvVar{{Name: "SERVING_READINESS_PROBE", Value: "{\"tcpSocket\":{\"port\":8080},\"timeoutSeconds\":1,\"periodSeconds\":10,\"successThreshold\":1,\"failureThreshold\":3}"}},
+						},
+					},
+				},
+			},
+			expected: &corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "deployment",
+					Annotations: map[string]string{
+						constants.LoggerInternalAnnotationKey:        "true",
+						constants.LoggerSinkUrlInternalAnnotationKey: "http://httpbin.org/",
+						constants.LoggerModeInternalAnnotationKey:    string(v1beta1.LogAll),
+					},
+				},
+				Spec: corev1.PodSpec{
+					Containers: []corev1.Container{
+						{
+							Name: "sklearn",
+							ReadinessProbe: &corev1.Probe{
+								ProbeHandler: corev1.ProbeHandler{
+									TCPSocket: &corev1.TCPSocketAction{
+										Port: intstr.IntOrString{
+											IntVal: 8080,
+										},
+									},
+								},
+								InitialDelaySeconds: 0,
+								TimeoutSeconds:      1,
+								PeriodSeconds:       10,
+								SuccessThreshold:    1,
+								FailureThreshold:    3,
+							},
+						},
+						{
+							Name: "queue-proxy",
+							Env:  []corev1.EnvVar{{Name: "SERVING_READINESS_PROBE", Value: "{\"tcpSocket\":{\"port\":8080},\"timeoutSeconds\":1,\"periodSeconds\":10,\"successThreshold\":1,\"failureThreshold\":3}"}},
+						},
+						{
+							Name:  constants.AgentContainerName,
+							Image: loggerConfig.Image,
+							Args: []string{
+								LoggerArgumentLogUrl,
+								"http://httpbin.org/",
+								LoggerArgumentSourceUri,
+								"deployment",
+								LoggerArgumentMode,
+								"all",
+								LoggerArgumentInferenceService,
+								"sklearn",
+								LoggerArgumentNamespace,
+								"default",
+								LoggerArgumentEndpoint,
+								"default",
+								LoggerArgumentComponent,
+								"predictor",
+								LoggerArgumentMetadataAnnotations,
+								"Foo=Bar,Fizz=Buzz",
+								LoggerArgumentTlsSkipVerify,
+								"false",
+								constants.AgentComponentPortArgName,
+								constants.InferenceServiceDefaultHttpPort,
+							},
+							Ports: []corev1.ContainerPort{
+								{
+									Name:          "agent-port",
+									ContainerPort: constants.InferenceServiceDefaultAgentPort,
+									Protocol:      "TCP",
+								},
+							},
+							Env:       []corev1.EnvVar{{Name: "SERVING_READINESS_PROBE", Value: "{\"tcpSocket\":{\"port\":8080},\"timeoutSeconds\":1,\"periodSeconds\":10,\"successThreshold\":1,\"failureThreshold\":3}"}},
+							Resources: agentResourceRequirement,
+							ReadinessProbe: &corev1.Probe{
+								ProbeHandler: corev1.ProbeHandler{
+									HTTPGet: &corev1.HTTPGetAction{
+										HTTPHeaders: []corev1.HTTPHeader{
+											{
+												Name:  "K-Network-Probe",
+												Value: "queue",
+											},
+										},
+										Port:   intstr.FromInt(9081),
+										Path:   "/",
+										Scheme: "HTTP",
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+		},
 		"DoNotAddLogger": {
 			original: &corev1.Pod{
 				ObjectMeta: metav1.ObjectMeta{
diff --git a/python/kserve/docs/V1beta1LoggerSpec.md b/python/kserve/docs/V1beta1LoggerSpec.md
index 025dbd4b598..4010485a456 100644
--- a/python/kserve/docs/V1beta1LoggerSpec.md
+++ b/python/kserve/docs/V1beta1LoggerSpec.md
@@ -4,6 +4,7 @@ LoggerSpec specifies optional payload logging available for all components
 ## Properties
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
+**metadata_annotations** | **list[str]** | Matched inference service annotations for propagating to inference logger cloud events. | [optional] 
 **metadata_headers** | **list[str]** | Matched metadata HTTP headers for propagating to inference logger cloud events. | [optional] 
 **mode** | **str** | Specifies the scope of the loggers. &lt;br /&gt; Valid values are: &lt;br /&gt; - \&quot;all\&quot; (default): log both request and response; &lt;br /&gt; - \&quot;request\&quot;: log only request; &lt;br /&gt; - \&quot;response\&quot;: log only response &lt;br /&gt; | [optional] 
 **url** | **str** | URL to send logging events | [optional] 
diff --git a/python/kserve/kserve/models/v1beta1_logger_spec.py b/python/kserve/kserve/models/v1beta1_logger_spec.py
index 6fa0fcddd31..9a69524caf8 100644
--- a/python/kserve/kserve/models/v1beta1_logger_spec.py
+++ b/python/kserve/kserve/models/v1beta1_logger_spec.py
@@ -47,28 +47,33 @@ class V1beta1LoggerSpec(object):
                             and the value is json key in definition.
     """
     openapi_types = {
+        'metadata_annotations': 'list[str]',
         'metadata_headers': 'list[str]',
         'mode': 'str',
         'url': 'str'
     }
 
     attribute_map = {
+        'metadata_annotations': 'metadataAnnotations',
         'metadata_headers': 'metadataHeaders',
         'mode': 'mode',
         'url': 'url'
     }
 
-    def __init__(self, metadata_headers=None, mode=None, url=None, local_vars_configuration=None):  # noqa: E501
+    def __init__(self, metadata_annotations=None, metadata_headers=None, mode=None, url=None, local_vars_configuration=None):  # noqa: E501
         """V1beta1LoggerSpec - a model defined in OpenAPI"""  # noqa: E501
         if local_vars_configuration is None:
             local_vars_configuration = Configuration()
         self.local_vars_configuration = local_vars_configuration
 
+        self._metadata_annotations = None
         self._metadata_headers = None
         self._mode = None
         self._url = None
         self.discriminator = None
 
+        if metadata_annotations is not None:
+            self.metadata_annotations = metadata_annotations
         if metadata_headers is not None:
             self.metadata_headers = metadata_headers
         if mode is not None:
@@ -76,6 +81,29 @@ def __init__(self, metadata_headers=None, mode=None, url=None, local_vars_config
         if url is not None:
             self.url = url
 
+    @property
+    def metadata_annotations(self):
+        """Gets the metadata_annotations of this V1beta1LoggerSpec.  # noqa: E501
+
+        Matched inference service annotations for propagating to inference logger cloud events.  # noqa: E501
+
+        :return: The metadata_annotations of this V1beta1LoggerSpec.  # noqa: E501
+        :rtype: list[str]
+        """
+        return self._metadata_annotations
+
+    @metadata_annotations.setter
+    def metadata_annotations(self, metadata_annotations):
+        """Sets the metadata_annotations of this V1beta1LoggerSpec.
+
+        Matched inference service annotations for propagating to inference logger cloud events.  # noqa: E501
+
+        :param metadata_annotations: The metadata_annotations of this V1beta1LoggerSpec.  # noqa: E501
+        :type: list[str]
+        """
+
+        self._metadata_annotations = metadata_annotations
+
     @property
     def metadata_headers(self):
         """Gets the metadata_headers of this V1beta1LoggerSpec.  # noqa: E501
diff --git a/test/crds/serving.kserve.io_inferenceservices.yaml b/test/crds/serving.kserve.io_inferenceservices.yaml
index 74045133923..9fe8571309d 100644
--- a/test/crds/serving.kserve.io_inferenceservices.yaml
+++ b/test/crds/serving.kserve.io_inferenceservices.yaml
@@ -8200,6 +8200,10 @@ spec:
                     type: object
                   logger:
                     properties:
+                      metadataAnnotations:
+                        items:
+                          type: string
+                        type: array
                       metadataHeaders:
                         items:
                           type: string
@@ -12710,6 +12714,10 @@ spec:
                     type: object
                   logger:
                     properties:
+                      metadataAnnotations:
+                        items:
+                          type: string
+                        type: array
                       metadataHeaders:
                         items:
                           type: string
@@ -25645,6 +25653,10 @@ spec:
                     type: object
                   logger:
                     properties:
+                      metadataAnnotations:
+                        items:
+                          type: string
+                        type: array
                       metadataHeaders:
                         items:
                           type: string


Reviews:
user: login: spolti
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T14:23:47Z
user: login: tylerhyang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T15:53:35Z
user: login: spolti
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:23:45Z
user: login: sivanantha321
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T06:14:13Z
user: login: sivanantha321
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T06:19:50Z
user: login: sivanantha321
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T06:30:47Z
user: login: sivanantha321
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T06:32:47Z
user: login: tylerhyang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T22:21:59Z
user: login: tylerhyang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T22:22:04Z
user: login: tylerhyang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T22:23:24Z
user: login: spolti
body: I am not sure about the amount of changes on this file:
[charts/kserve-crd/templates/serving.kserve.io_inferenceservices.yaml](https://github.com/kserve/kserve/pull/4325/files#diff-a46255812489769b7945a0f72b7d0550580e9c3bffa3d9126b3df36a0a96d950)


can you please check?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T14:31:08Z
user: login: yuzisun
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-13T16:51:42Z
user: login: yuzisun
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-13T16:52:46Z
user: login: tylerhyang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T17:13:25Z
user: login: tylerhyang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T17:18:12Z
user: login: yuzisun
body: Thanks @tylerhyang ! nice work
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-18T22:18:00Z

Review comments:
diff_hunk: @@ -58,15 +59,16 @@ var (
 	configDir    = flag.String("config-dir", "/mnt/configs", "directory for model config files")
 	modelDir     = flag.String("model-dir", "/mnt/models", "directory for model files")
 	// logger flags
-	logUrl           = flag.String("log-url", "", "The URL to send request/response logs to")
-	workers          = flag.Int("workers", 5, "Number of workers")
-	sourceUri        = flag.String("source-uri", "", "The source URI to use when publishing cloudevents")
-	logMode          = flag.String("log-mode", string(v1beta1.LogAll), "Whether to log 'request', 'response' or 'all'")
-	inferenceService = flag.String("inference-service", "", "The InferenceService name to add as header to log events")
-	namespace        = flag.String("namespace", "", "The namespace to add as header to log events")
-	endpoint         = flag.String("endpoint", "", "The endpoint name to add as header to log events")
-	component        = flag.String("component", "", "The component name (predictor, explainer, transformer) to add as header to log events")
-	metadataHeaders  = flag.StringSlice("metadata-headers", nil, "Allow list of headers that will be passed down as metadata")
+	logUrl              = flag.String("log-url", "", "The URL to send request/response logs to")
+	workers             = flag.Int("workers", 5, "Number of workers")
+	sourceUri           = flag.String("source-uri", "", "The source URI to use when publishing cloudevents")
+	logMode             = flag.String("log-mode", string(v1beta1.LogAll), "Whether to log 'request', 'response' or 'all'")
+	inferenceService    = flag.String("inference-service", "", "The InferenceService name to add as header to log events")
+	namespace           = flag.String("namespace", "", "The namespace to add as header to log events")
+	endpoint            = flag.String("endpoint", "", "The endpoint name to add as header to log events")
+	component           = flag.String("component", "", "The component name (predictor, explainer, transformer) to add as header to log events")
+	metadataHeaders     = flag.StringSlice("metadata-headers", nil, "Allow list of headers that will be passed down as metadata")
+	metadataAnnotations = flag.StringSlice("metadata-annotations", nil, "pod metadata annotation to be passed with payload logging")
path: cmd/agent/main.go
user: login: spolti
body: ```suggestion
	metadataAnnotations = flag.StringSlice("metadata-annotations", nil, "Allow list of metadata annotation to be passed with payload logging")
```
created_at: 2025-03-21T14:23:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,15 +59,16 @@ var (
 	configDir    = flag.String("config-dir", "/mnt/configs", "directory for model config files")
 	modelDir     = flag.String("model-dir", "/mnt/models", "directory for model files")
 	// logger flags
-	logUrl           = flag.String("log-url", "", "The URL to send request/response logs to")
-	workers          = flag.Int("workers", 5, "Number of workers")
-	sourceUri        = flag.String("source-uri", "", "The source URI to use when publishing cloudevents")
-	logMode          = flag.String("log-mode", string(v1beta1.LogAll), "Whether to log 'request', 'response' or 'all'")
-	inferenceService = flag.String("inference-service", "", "The InferenceService name to add as header to log events")
-	namespace        = flag.String("namespace", "", "The namespace to add as header to log events")
-	endpoint         = flag.String("endpoint", "", "The endpoint name to add as header to log events")
-	component        = flag.String("component", "", "The component name (predictor, explainer, transformer) to add as header to log events")
-	metadataHeaders  = flag.StringSlice("metadata-headers", nil, "Allow list of headers that will be passed down as metadata")
+	logUrl              = flag.String("log-url", "", "The URL to send request/response logs to")
+	workers             = flag.Int("workers", 5, "Number of workers")
+	sourceUri           = flag.String("source-uri", "", "The source URI to use when publishing cloudevents")
+	logMode             = flag.String("log-mode", string(v1beta1.LogAll), "Whether to log 'request', 'response' or 'all'")
+	inferenceService    = flag.String("inference-service", "", "The InferenceService name to add as header to log events")
+	namespace           = flag.String("namespace", "", "The namespace to add as header to log events")
+	endpoint            = flag.String("endpoint", "", "The endpoint name to add as header to log events")
+	component           = flag.String("component", "", "The component name (predictor, explainer, transformer) to add as header to log events")
+	metadataHeaders     = flag.StringSlice("metadata-headers", nil, "Allow list of headers that will be passed down as metadata")
+	metadataAnnotations = flag.StringSlice("metadata-annotations", nil, "pod metadata annotation to be passed with payload logging")
path: cmd/agent/main.go
user: login: tylerhyang
body: thanks for the suggestion!
created_at: 2025-03-21T15:53:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -286,6 +289,14 @@ func startLogger(workers int, logger *zap.SugaredLogger) *loggerArgs {
 		os.Exit(-1)
 	}
 
+	var annotationKVPair map[string]string = map[string]string{}
+	for _, annotations := range *metadataAnnotations {
+		k, v, found := strings.Cut(annotations, "=")
+		if found {
+			annotationKVPair[k] = v
path: cmd/agent/main.go
user: login: sivanantha321
body: Can we do a log if the specified annotation is not conforming to the format ?
created_at: 2025-03-28T06:14:13Z
author_association: MEMBER
diff_hunk: @@ -2734,6 +2734,10 @@ spec:
                       type: object
                     logger:
                       properties:
+                        metadataAnnotations:
path: config/crd/full/serving.kserve.io_inferenceservices.yaml
user: login: sivanantha321
body: Can you help update the helm chart CRD as well ?
https://github.com/kserve/kserve/blob/master/charts/kserve-crd/templates/serving.kserve.io_inferenceservices.yaml
created_at: 2025-03-28T06:19:50Z
author_association: MEMBER
diff_hunk: @@ -133,6 +134,15 @@ func (w *Worker) sendCloudEvent(logReq LogRequest) error {
 	}
 	event.SetExtension(MetadataAttr, string(encodedMetadata))
 
+	if len(logReq.Annotations) > 0 {
+		bits, err := json.Marshal(logReq.Annotations)
+		if err != nil {
+			fmt.Println("failed to marshal annotations")
path: pkg/logger/worker.go
user: login: sivanantha321
body: We can use logger
created_at: 2025-03-28T06:30:47Z
author_association: MEMBER
diff_hunk: @@ -212,6 +213,18 @@ func (ag *AgentInjector) InjectAgent(pod *corev1.Pod) error {
 			loggerArgs = append(loggerArgs, LoggerArgumentMetadataHeaders)
 			loggerArgs = append(loggerArgs, logHeaderMetadata)
 		}
+		logMetadataAnnotations, ok := pod.ObjectMeta.Annotations[constants.LoggerMetadataAnnotationsInternalAnnotationKey]
+		if ok {
path: pkg/webhook/admission/pod/agent_injector.go
user: login: sivanantha321
body: Can we do log if the specified annotation is not exist ?
created_at: 2025-03-28T06:32:47Z
author_association: MEMBER
diff_hunk: @@ -286,6 +289,14 @@ func startLogger(workers int, logger *zap.SugaredLogger) *loggerArgs {
 		os.Exit(-1)
 	}
 
+	var annotationKVPair map[string]string = map[string]string{}
+	for _, annotations := range *metadataAnnotations {
+		k, v, found := strings.Cut(annotations, "=")
+		if found {
+			annotationKVPair[k] = v
path: cmd/agent/main.go
user: login: tylerhyang
body: sure; added a warning log if the format mismatches
created_at: 2025-03-28T22:21:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2734,6 +2734,10 @@ spec:
                       type: object
                     logger:
                       properties:
+                        metadataAnnotations:
path: config/crd/full/serving.kserve.io_inferenceservices.yaml
user: login: tylerhyang
body: done
created_at: 2025-03-28T22:22:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -133,6 +134,15 @@ func (w *Worker) sendCloudEvent(logReq LogRequest) error {
 	}
 	event.SetExtension(MetadataAttr, string(encodedMetadata))
 
+	if len(logReq.Annotations) > 0 {
+		bits, err := json.Marshal(logReq.Annotations)
+		if err != nil {
+			fmt.Println("failed to marshal annotations")
path: pkg/logger/worker.go
user: login: tylerhyang
body: changed to logger
created_at: 2025-03-28T22:23:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -286,6 +289,16 @@ func startLogger(workers int, logger *zap.SugaredLogger) *loggerArgs {
 		os.Exit(-1)
 	}
 
+	var annotationKVPair map[string]string = map[string]string{}
+	for _, annotations := range *metadataAnnotations {
+		k, v, found := strings.Cut(annotations, "=")
+		if found {
+			annotationKVPair[k] = v
+		} else {
+			logger.Warnf("annotation does not adhere to desired format got key: %s value: %s", k, v)
path: cmd/agent/main.go
user: login: yuzisun
body: I think it is better to error out and exit the agent if the annotation is not specified correctly instead of silently printing out the warnings.
created_at: 2025-04-13T16:51:42Z
author_association: MEMBER
diff_hunk: @@ -133,6 +134,15 @@ func (w *Worker) sendCloudEvent(logReq LogRequest) error {
 	}
 	event.SetExtension(MetadataAttr, string(encodedMetadata))
 
+	if len(logReq.Annotations) > 0 {
+		bits, err := json.Marshal(logReq.Annotations)
+		if err != nil {
+			w.Log.Info("failed to marshal annotations")
path: pkg/logger/worker.go
user: login: yuzisun
body: Should print error 
created_at: 2025-04-13T16:52:46Z
author_association: MEMBER
diff_hunk: @@ -133,6 +134,15 @@ func (w *Worker) sendCloudEvent(logReq LogRequest) error {
 	}
 	event.SetExtension(MetadataAttr, string(encodedMetadata))
 
+	if len(logReq.Annotations) > 0 {
+		bits, err := json.Marshal(logReq.Annotations)
+		if err != nil {
+			w.Log.Info("failed to marshal annotations")
path: pkg/logger/worker.go
user: login: tylerhyang
body: included returned error
created_at: 2025-04-14T17:13:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -286,6 +289,16 @@ func startLogger(workers int, logger *zap.SugaredLogger) *loggerArgs {
 		os.Exit(-1)
 	}
 
+	var annotationKVPair map[string]string = map[string]string{}
+	for _, annotations := range *metadataAnnotations {
+		k, v, found := strings.Cut(annotations, "=")
+		if found {
+			annotationKVPair[k] = v
+		} else {
+			logger.Warnf("annotation does not adhere to desired format got key: %s value: %s", k, v)
path: cmd/agent/main.go
user: login: tylerhyang
body: done; agent now exits when a malformed annotation pair is found

created_at: 2025-04-14T17:18:12Z
author_association: CONTRIBUTOR

PR comments:
user: login: tylerhyang
created_at: 2025-03-21T05:47:46Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: spolti
created_at: 2025-03-21T14:00:47Z
body: Hi Tyler, thanks for sending this.
Can you please help me to understand why passing the annotations to the agent logger is necessary?

author_association: CONTRIBUTOR
user: login: spolti
created_at: 2025-03-21T14:22:59Z
body: Nevermind, just saw https://github.com/kserve/website/pull/458
author_association: CONTRIBUTOR
user: login: spolti
created_at: 2025-03-21T14:48:03Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: tylerhyang
created_at: 2025-03-25T05:22:51Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: spolti
created_at: 2025-03-31T13:15:37Z
body: Hi @tylerhyang, can you please fix the conflicts?
author_association: CONTRIBUTOR
user: login: tylerhyang
created_at: 2025-04-01T22:12:40Z
body: > Hi @tylerhyang, can you please fix the conflicts?

Hey, I regenerated the manifests with `make manifests` with the helm chart generation code uncommented (lines 112 and 113 in Makefile ) but the PR diff looks suspiciously large (>19k).  Can someone confirm that this is the proper way to regenerate the helm CRDs?
cc: @sivanantha321 @spolti 
author_association: CONTRIBUTOR
user: login: tylerhyang
created_at: 2025-04-04T17:37:32Z
body: /rerun-all
author_association: CONTRIBUTOR
user: login: tylerhyang
created_at: 2025-04-10T16:53:58Z
body: @spolti updated with the proper CRD generation.  Please take a look now :)
author_association: CONTRIBUTOR