Repository: kserve/kserve

Issue:
KServe CloudEvents infinite-loop using Knative Eventing Broker
/kind bug

**What steps did you take and what happened:**
We're using Knative eventing `Broker` and `Trigger` with a filter for triggering the `PredictHandler` handler in a custom model server using KServe python library.
Currently, KServe python library sends the response CloudEvent with the same attributes from the original CloudEvent and there is no option to filter the response CloudEvent to avoid the **infinite-loop** of CloudEvents being sent to KServe from the `Broker`.



**What did you expect to happen:**
We expect to at least change the response CloudEvent `source` field to KServe instance name or some env variable.

**Anything else you would like to add:**
Line of problematic code: https://github.com/kserve/kserve/blob/464308defb7245110a3dbf255629d7f418bb9947/python/kserve/kserve/handlers/http.py#L99


**Environment:**

- Istio Version: 1.11.4
- Knative Version: 1.0
- KFServing Version: 0.7
- Kubeflow version: N/A
- Kfdef: N/A
- Minikube version: N/A
- Kubernetes version: (use `kubectl version`): 1.21.2
- OS (e.g. from `/etc/os-release`): Amazon Linux 2
Cloud events responses need to be configurable for custom predictors and transformers
/kind feature

**Describe the solution you'd like**
[A clear and concise description of what you want to happen.]

Cloud events responses need to be configurable for custom predictors and transformers. Currently it returns the same Ce-Type and Ce-Source, which means that the transformer/predictor will get those events again except this time without the proper payload, at any case they need to be configurable so the proper consumer can pick them up. 


**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]

I already have an example how this should be done but I am unsure how to allow for configuration view the custom spec. Should this be env variables? 

https://github.com/pwc-labs/kfserving/blob/ce-time-fix/python/kfserving/kfserving/handlers/http.py#L85

Also note that: RFC 3339 is not compatible with ISO. I corrected this as well. 

```python
self.set_header('ce-time', datetime.utcnow().replace(tzinfo=pytz.utc).trftime('%Y-%m-%dT%H:%M:%SZ'))
````


Discussion:
I've been using Knative Eventing and CloudEvents recently so I will take a look at this.

Could you give me an example of your `Trigger`? Are you filtering based on event type attribute?

And am I understanding correctly, your system looks something like this:

System A produces a CloudEvent (Event A) -> Broker -> Trigger -> KServe Model Server

And the response from KServe Model Server goes into the Broker as well?
And because KServe Model Server uses same attributes as original CloudEvent Event A, it is sent again by the Trigger

Is System A that produces the original event also a KServe Model Server?

If I understand correctly, then as you said just setting the source field would probably be enough.
> I've been using Knative Eventing and CloudEvents recently so I will take a look at this.
> 
> Could you give me an example of your `Trigger`? Are you filtering based on event type attribute?
> 
> And am I understanding correctly, your system looks something like this:
> 
> System A produces a CloudEvent (Event A) -> Broker -> Trigger -> KServe Model Server
> 
> And the response from KServe Model Server goes into the Broker as well? And because KServe Model Server uses the same attributes as original CloudEvent Event A, it is sent again by the Trigger
> 
> Is System A that produces the original event also a KServe Model Server?
> 
> If I understand correctly, then as you said just setting the source field would probably be enough.

1. Yes, I'm filtering based on the event type attribute. I would like to use the source attribute as well to differentiate between request and response, but as I've mentioned it's not implemented that way right now.
2. You're right, the response goes to the Broker as part of the request-response option of Knative eventing (without a trigger for the response).
3. However, currently we're making a request directly to KServe Model Server in order to avoid the problem, and just using a Trigger for `org.kubeflow.serving.inference.response` for another service that recieves the model response.

I didn't mention it, but our process takes a couple of minutes to complete, so it's vital to be able to run KServe inferencing in an async way.

4. System A is not a KServe Model Server.

If we could send the request to the Broker, use a Trigger with a destination of a KServe Model Server and then get the response back or keep using another Trigger with the logger model response it would be more ideal.
@maganaluis ye I think env variable is fine, if not present we can set the default type/source as you suggested

Resolving patch:
diff --git a/python/kserve/kserve/handlers/http.py b/python/kserve/kserve/handlers/http.py
--- a/python/kserve/kserve/handlers/http.py
+++ b/python/kserve/kserve/handlers/http.py
@@ -10,19 +10,17 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
 from typing import Any
 
 import tornado.web
 import json
-import pytz
 import cloudevents.exceptions as ce
-from cloudevents.http import CloudEvent, from_http, is_binary, is_structured, to_binary, to_structured
+from cloudevents.http import CloudEvent, from_http
 from cloudevents.sdk.converters.util import has_binary_headers
 from http import HTTPStatus
 from kserve.kfmodel_repository import KFModelRepository
 from kserve.kfmodel import ModelType
-from datetime import datetime
+from kserve.utils.utils import is_structured_cloudevent, create_response_cloudevent
 
 from ray.serve.api import RayServeHandle
 
@@ -65,20 +63,30 @@ def get_model(self, name: str):
 
 
 class PredictHandler(HTTPHandler):
+    def get_binary_cloudevent(self) -> CloudEvent:
+        try:
+            # Use default unmarshaller if contenttype is set in header
+            if "ce-contenttype" in self.request.headers:
+                event = from_http(self.request.headers, self.request.body)
+            else:
+                event = from_http(self.request.headers, self.request.body, lambda x: x)
+
+            return event
+        except (ce.MissingRequiredFields, ce.InvalidRequiredFields, ce.InvalidStructuredJSON,
+                ce.InvalidHeadersFormat, ce.DataMarshallerError, ce.DataUnmarshallerError) as e:
+            raise tornado.web.HTTPError(
+                status_code=HTTPStatus.BAD_REQUEST,
+                reason="Cloud Event Exceptions: %s" % e
+            )
+
     async def post(self, name: str):
+        is_cloudevent = False
+        is_binary_cloudevent = False
+
         if has_binary_headers(self.request.headers):
-            try:
-                # Use default unmarshaller if contenttype is set in header
-                if "ce-contenttype" in self.request.headers:
-                    body = from_http(self.request.headers, self.request.body)
-                else:
-                    body = from_http(self.request.headers, self.request.body, lambda x: x)
-            except (ce.MissingRequiredFields, ce.InvalidRequiredFields, ce.InvalidStructuredJSON,
-                    ce.InvalidHeadersFormat, ce.DataMarshallerError, ce.DataUnmarshallerError) as e:
-                raise tornado.web.HTTPError(
-                    status_code=HTTPStatus.BAD_REQUEST,
-                    reason="Cloud Event Exceptions: %s" % e
-                )
+            is_cloudevent = True
+            is_binary_cloudevent = True
+            body = self.get_binary_cloudevent()
         else:
             try:
                 body = json.loads(self.request.body)
@@ -87,6 +95,10 @@ async def post(self, name: str):
                     status_code=HTTPStatus.BAD_REQUEST,
                     reason="Unrecognized request format: %s" % e
                 )
+
+            if is_structured_cloudevent(body):
+                is_cloudevent = True
+
         # call model locally or remote model workers
         model = self.get_model(name)
         if not isinstance(model, RayServeHandle):
@@ -94,20 +106,18 @@ async def post(self, name: str):
         else:
             model_handle = model
             response = await model_handle.remote(body)
-        # process response from the model
-        if has_binary_headers(self.request.headers):
-            event = CloudEvent(body._attributes, response)
-            if is_binary(self.request.headers):
-                eventheader, eventbody = to_binary(event)
-            elif is_structured(self.request.headers):
-                eventheader, eventbody = to_structured(event)
-            for k, v in eventheader.items():
-                if k != "ce-time":
-                    self.set_header(k, v)
-                else:  # utc now() timestamp
-                    self.set_header('ce-time', datetime.utcnow().replace(tzinfo=pytz.utc).
-                                    strftime('%Y-%m-%dT%H:%M:%S.%f%z'))
-            response = eventbody
+
+        # if we received a cloudevent, then also return a cloudevent
+        if is_cloudevent:
+            headers, response = create_response_cloudevent(name, body, response, is_binary_cloudevent)
+
+            if is_binary_cloudevent:
+                self.set_header("Content-Type", "application/json")
+            else:
+                self.set_header("Content-Type", "application/cloudevents+json")
+
+            for k, v in headers.items():
+                self.set_header(k, v)
 
         self.write(response)
 
diff --git a/python/kserve/kserve/kfmodel.py b/python/kserve/kserve/kfmodel.py
--- a/python/kserve/kserve/kfmodel.py
+++ b/python/kserve/kserve/kfmodel.py
@@ -20,10 +20,12 @@
 from cloudevents.http import CloudEvent
 from http import HTTPStatus
 from enum import Enum
+from kserve.utils.utils import is_structured_cloudevent
 import grpc
 from tritonclient.grpc import InferResult, service_pb2_grpc
 from tritonclient.grpc.service_pb2 import ModelInferRequest, ModelInferResponse
 
+
 PREDICTOR_URL_FORMAT = "http://{0}/v1/models/{1}:predict"
 EXPLAINER_URL_FORMAT = "http://{0}/v1/models/{1}:explain"
 PREDICTOR_V2_URL_FORMAT = "http://{0}/v2/models/{1}/infer"
@@ -108,37 +110,35 @@ def load(self) -> bool:
         self.ready = True
         return self.ready
 
-    async def preprocess(self, request: Dict) -> Union[Dict, ModelInferRequest]:
+    async def preprocess(self, request: Union[Dict, CloudEvent]) -> Union[Dict, ModelInferRequest]:
         """
-        The preprocess handler can be overridden for data or feature transformation,
-        the default implementation decodes to Dict if it is cloudevent JSON otherwise pass the data field
-        :param request: JSON Dict or CloudEvent
+        The preprocess handler can be overridden for data or feature transformation.
+        The default implementation decodes to Dict if it is a binary CloudEvent
+        or gets the data field from a structured CloudEvent.
+        :param request: Dict|CloudEvent|ModelInferRequest
         :return: Transformed Dict|ModelInferRequest which passes to predict handler
         """
         response = request
 
         if isinstance(request, CloudEvent):
             response = request.data
-            if isinstance(response, bytes):
-                try:
-                    response = json.loads(response.decode('UTF-8'))
-                except (json.decoder.JSONDecodeError, UnicodeDecodeError) as e:
-                    attributes = request._attributes
-                    if "content-type" in attributes:
-                        if attributes["content-type"] == "application/cloudevents+json" or \
-                           attributes["content-type"] == "application/json":
-                            raise tornado.web.HTTPError(
-                                status_code=HTTPStatus.BAD_REQUEST,
-                                reason="Unrecognized request format: %s" % e
-                            )
-        elif isinstance(request, dict):
+            # Try to decode and parse JSON UTF-8 if possible, otherwise
+            # just pass the CloudEvent data on to the predict function.
+            # This is for the cases that CloudEvent encoding is protobuf, avro etc.
+            try:
+                response = json.loads(response.decode('UTF-8'))
+            except (json.decoder.JSONDecodeError, UnicodeDecodeError) as e:
+                # If decoding or parsing failed, check if it was supposed to be JSON UTF-8
+                if "content-type" in request._attributes and \
+                        (request._attributes["content-type"] == "application/cloudevents+json" or
+                         request._attributes["content-type"] == "application/json"):
+                    raise tornado.web.HTTPError(
+                        status_code=HTTPStatus.BAD_REQUEST,
+                        reason=f"Failed to decode or parse binary json cloudevent: {e}"
+                    )
 
-            if "time" in request \
-                    and "type" in request \
-                    and "source" in request \
-                    and "id" in request \
-                    and "specversion" in request \
-                    and "data" in request:
+        elif isinstance(request, dict):
+            if is_structured_cloudevent(request):
                 response = request["data"]
 
         return response
diff --git a/python/kserve/kserve/utils/utils.py b/python/kserve/kserve/utils/utils.py
--- a/python/kserve/kserve/utils/utils.py
+++ b/python/kserve/kserve/utils/utils.py
@@ -14,6 +14,8 @@
 import os
 import sys
 import psutil
+from typing import Dict, Union
+from cloudevents.http import CloudEvent, to_binary, to_structured
 
 
 def is_running_in_k8s():
@@ -68,3 +70,43 @@ def cpu_count():
             pass
 
     return count
+
+
+def is_structured_cloudevent(body: Dict) -> bool:
+    """Returns True if the JSON request body resembles a structured CloudEvent"""
+    return "time" in body \
+           and "type" in body \
+           and "source" in body \
+           and "id" in body \
+           and "specversion" in body \
+           and "data" in body
+
+
+def create_response_cloudevent(model_name: str, body: Union[Dict, CloudEvent], response: Dict,
+                               binary_event=False) -> tuple:
+    ce_attributes = {}
+
+    if os.getenv("CE_MERGE", "false").lower() == "true":
+        if binary_event:
+            ce_attributes = body._attributes
+            if "datacontenttype" in ce_attributes:  # Optional field so must check
+                del ce_attributes["datacontenttype"]
+        else:
+            ce_attributes = body
+            del ce_attributes["data"]
+
+        # Remove these fields so we generate new ones
+        del ce_attributes["id"]
+        del ce_attributes["time"]
+
+    ce_attributes["type"] = os.getenv("CE_TYPE", "io.kserve.inference.response")
+    ce_attributes["source"] = os.getenv("CE_SOURCE", f"io.kserve.kfserver.{model_name}")
+
+    event = CloudEvent(ce_attributes, response)
+
+    if binary_event:
+        event_headers, event_body = to_binary(event)
+    else:
+        event_headers, event_body = to_structured(event)
+
+    return event_headers, event_body


Test patch:
diff --git a/python/kserve/test/test_server.py b/python/kserve/test/test_server.py
--- a/python/kserve/test/test_server.py
+++ b/python/kserve/test/test_server.py
@@ -11,7 +11,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import json
+import os
 import re
+from unittest import mock
 
 import avro.io
 import avro.schema
@@ -39,7 +41,7 @@
         '''
 
 
-def dummy_cloud_event(data, set_contenttype=False):
+def dummy_cloud_event(data, set_contenttype=False, add_extension=False):
     # This data defines a binary cloudevent
     attributes = {
         "type": "com.example.sampletype1",
@@ -50,6 +52,8 @@ def dummy_cloud_event(data, set_contenttype=False):
     }
     if set_contenttype:
         attributes["content-type"] = "application/json"
+    if add_extension:
+        attributes["custom-extension"] = "custom-value"
 
     event = CloudEvent(attributes, data)
     return event
@@ -130,15 +134,13 @@ def preprocess(self, request):
             assert attributes["type"] == "com.example.sampletype1"
             assert attributes["datacontenttype"] == "application/x-www-form-urlencoded"
             assert attributes["content-type"] == "application/json"
-            return request.data
+            return self._parserequest(request.data)
 
     async def predict(self, request):
-        record1 = self._parserequest(request)
-        return {"predictions": [[record1['name'], record1['favorite_number'], record1['favorite_color']]]}
+        return {"predictions": [[request['name'], request['favorite_number'], request['favorite_color']]]}
 
     async def explain(self, request):
-        record1 = self._parserequest(request)
-        return {"predictions": [[record1['name'], record1['favorite_number'], record1['favorite_color']]]}
+        return {"predictions": [[request['name'], request['favorite_number'], request['favorite_color']]]}
 
 
 class DummyKFModelRepository(KFModelRepository):
@@ -189,19 +191,6 @@ async def test_predict(self, http_server_client):
         assert resp.body == b'{"predictions": [[1, 2]]}'
         assert resp.headers['content-type'] == "application/json; charset=UTF-8"
 
-    async def test_predict_ce_structured(self, http_server_client):
-
-        event = dummy_cloud_event({"instances": [[1, 2]]})
-        headers, body = to_structured(event)
-        resp = await http_server_client.fetch('/v1/models/TestModel:predict',
-                                              method="POST",
-                                              headers=headers,
-                                              body=body)
-
-        assert resp.code == 200
-        assert resp.body == b'{"predictions": [[1, 2]]}'
-        assert resp.headers['content-type'] == "application/json; charset=UTF-8"
-
     async def test_explain(self, http_server_client):
         resp = await http_server_client.fetch('/v1/models/TestModel:explain',
                                               method="POST",
@@ -322,7 +311,6 @@ async def test_model_not_ready_error(self, http_server_client):
 
 
 class TestTFHttpServerCloudEvent:
-
     @pytest.fixture(scope="class")
     def app(self):  # pylint: disable=no-self-use
         model = DummyCEModel("TestModel")
@@ -330,6 +318,88 @@ def app(self):  # pylint: disable=no-self-use
         server.register_model(model)
         return server.create_application()
 
+    async def test_predict_ce_structured(self, http_server_client):
+        event = dummy_cloud_event({"instances": [[1, 2]]})
+        headers, body = to_structured(event)
+
+        resp = await http_server_client.fetch('/v1/models/TestModel:predict',
+                                              method="POST",
+                                              headers=headers,
+                                              body=body)
+        body = json.loads(resp.body)
+
+        assert resp.code == 200
+        assert resp.headers['content-type'] == "application/cloudevents+json"
+
+        assert body["id"] != "36077800-0c23-4f38-a0b4-01f4369f670a"
+        assert body["data"] == {"predictions": [[1, 2]]}
+        assert body['specversion'] == "1.0"
+        assert body['source'] == "io.kserve.kfserver.TestModel"
+        assert body['type'] == "io.kserve.inference.response"
+        assert body['time'] > "2021-01-28T21:04:43.144141+00:00"
+
+    async def test_predict_custom_ce_attributes(self, http_server_client):
+        with mock.patch.dict(os.environ,
+                             {"CE_SOURCE": "io.kserve.kfserver.CustomSource", "CE_TYPE": "io.kserve.custom_type"}):
+            event = dummy_cloud_event({"instances": [[1, 2]]})
+            headers, body = to_structured(event)
+
+            resp = await http_server_client.fetch('/v1/models/TestModel:predict',
+                                                  method="POST",
+                                                  headers=headers,
+                                                  body=body)
+            body = json.loads(resp.body)
+
+            assert resp.code == 200
+            assert resp.headers['content-type'] == "application/cloudevents+json"
+
+            assert body["id"] != "36077800-0c23-4f38-a0b4-01f4369f670a"
+            assert body["data"] == {"predictions": [[1, 2]]}
+            assert body['source'] == "io.kserve.kfserver.CustomSource"
+            assert body['type'] == "io.kserve.custom_type"
+
+    async def test_predict_merge_structured_ce_attributes(self, http_server_client):
+        with mock.patch.dict(os.environ, {"CE_MERGE": "true"}):
+            event = dummy_cloud_event({"instances": [[1, 2]]}, add_extension=True)
+            headers, body = to_structured(event)
+
+            resp = await http_server_client.fetch('/v1/models/TestModel:predict',
+                                                  method="POST",
+                                                  headers=headers,
+                                                  body=body)
+            body = json.loads(resp.body)
+
+            assert resp.code == 200
+            assert resp.headers['content-type'] == "application/cloudevents+json"
+
+            assert body["id"] != "36077800-0c23-4f38-a0b4-01f4369f670a"
+            assert body["data"] == {"predictions": [[1, 2]]}
+            assert body['source'] == "io.kserve.kfserver.TestModel"
+            assert body['type'] == "io.kserve.inference.response"
+            assert body["custom-extension"] == "custom-value"  # Added by add_extension=True in dummy_cloud_event
+            assert body['time'] > "2021-01-28T21:04:43.144141+00:00"
+
+    async def test_predict_merge_binary_ce_attributes(self, http_server_client):
+        with mock.patch.dict(os.environ, {"CE_MERGE": "true"}):
+            event = dummy_cloud_event({"instances": [[1, 2]]}, set_contenttype=True, add_extension=True)
+            headers, body = to_binary(event)
+
+            resp = await http_server_client.fetch('/v1/models/TestModel:predict',
+                                                  method="POST",
+                                                  headers=headers,
+                                                  body=body)
+
+            assert resp.code == 200
+            assert resp.headers['content-type'] == "application/json"
+            assert resp.headers['ce-specversion'] == "1.0"
+            assert resp.headers["ce-id"] != "36077800-0c23-4f38-a0b4-01f4369f670a"
+            # Added by add_extension=True in dummy_cloud_event
+            assert resp.headers['ce-custom-extension'] == 'custom-value'
+            assert resp.headers['ce-source'] == "io.kserve.kfserver.TestModel"
+            assert resp.headers['ce-type'] == "io.kserve.inference.response"
+            assert resp.headers['ce-time'] > "2021-01-28T21:04:43.144141+00:00"
+            assert resp.body == b'{"predictions": [[1, 2]]}'
+
     async def test_predict_ce_binary_dict(self, http_server_client):
         event = dummy_cloud_event({"instances": [[1, 2]]}, set_contenttype=True)
         headers, body = to_binary(event)
@@ -339,14 +409,13 @@ async def test_predict_ce_binary_dict(self, http_server_client):
                                               body=body)
 
         assert resp.code == 200
-        assert resp.body == b'{"predictions": [[1, 2]]}'
-        assert resp.headers['content-type'] == "application/x-www-form-urlencoded"
+        assert resp.headers['content-type'] == "application/json"
         assert resp.headers['ce-specversion'] == "1.0"
-        assert resp.headers['ce-id'] == "36077800-0c23-4f38-a0b4-01f4369f670a"
-        assert resp.headers['ce-source'] == "https://example.com/event-producer"
-        assert resp.headers['ce-type'] == "com.example.sampletype1"
-        assert resp.headers['ce-datacontenttype'] == "application/x-www-form-urlencoded"
+        assert resp.headers["ce-id"] != "36077800-0c23-4f38-a0b4-01f4369f670a"
+        assert resp.headers['ce-source'] == "io.kserve.kfserver.TestModel"
+        assert resp.headers['ce-type'] == "io.kserve.inference.response"
         assert resp.headers['ce-time'] > "2021-01-28T21:04:43.144141+00:00"
+        assert resp.body == b'{"predictions": [[1, 2]]}'
 
     async def test_predict_ce_binary_bytes(self, http_server_client):
         event = dummy_cloud_event(b'{"instances":[[1,2]]}', set_contenttype=True)
@@ -357,14 +426,13 @@ async def test_predict_ce_binary_bytes(self, http_server_client):
                                               body=body)
 
         assert resp.code == 200
-        assert resp.body == b'{"predictions": [[1, 2]]}'
-        assert resp.headers['content-type'] == "application/x-www-form-urlencoded"
+        assert resp.headers['content-type'] == "application/json"
         assert resp.headers['ce-specversion'] == "1.0"
-        assert resp.headers['ce-id'] == "36077800-0c23-4f38-a0b4-01f4369f670a"
-        assert resp.headers['ce-source'] == "https://example.com/event-producer"
-        assert resp.headers['ce-type'] == "com.example.sampletype1"
-        assert resp.headers['ce-datacontenttype'] == "application/x-www-form-urlencoded"
+        assert resp.headers["ce-id"] != "36077800-0c23-4f38-a0b4-01f4369f670a"
+        assert resp.headers['ce-source'] == "io.kserve.kfserver.TestModel"
+        assert resp.headers['ce-type'] == "io.kserve.inference.response"
         assert resp.headers['ce-time'] > "2021-01-28T21:04:43.144141+00:00"
+        assert resp.body == b'{"predictions": [[1, 2]]}'
 
     async def test_predict_ce_bytes_bad_format_exception(self, http_server_client):
         event = dummy_cloud_event(b'{', set_contenttype=True)
@@ -376,8 +444,8 @@ async def test_predict_ce_bytes_bad_format_exception(self, http_server_client):
                                                headers=headers,
                                                body=body)
         assert err.value.code == 400
-
-        error_regex = re.compile("Unrecognized request format: Expecting property name enclosed in double quotes.*")
+        error_regex = re.compile("Failed to decode or parse binary json cloudevent: "
+                                 "Expecting property name enclosed in double quotes.*")
         response = json.loads(err.value.response.body)
         assert error_regex.match(response["error"]) is not None
 
@@ -391,9 +459,8 @@ async def test_predict_ce_bytes_bad_hex_format_exception(self, http_server_clien
                                                headers=headers,
                                                body=body)
         assert err.value.code == 400
-
-        error_regex = re.compile("Unrecognized request format: 'utf-8' codec can't decode byte 0x80 in position 1: "
-                                 "invalid start byte.*")
+        error_regex = re.compile("Failed to decode or parse binary json cloudevent: "
+                                 "'utf-8' codec can't decode byte 0x80 in position 1: invalid start byte.*")
         response = json.loads(err.value.response.body)
         assert error_regex.match(response["error"]) is not None
 
@@ -418,7 +485,6 @@ async def test_predict_ce_avro_binary(self, http_server_client):
         data = bytes_writer.getvalue()
 
         event = dummy_cloud_event(data, set_contenttype=True)
-
         # Creates the HTTP request representation of the CloudEvent in binary content mode
         headers, body = to_binary(event)
         resp = await http_server_client.fetch('/v1/models/TestModel:predict',
@@ -427,11 +493,10 @@ async def test_predict_ce_avro_binary(self, http_server_client):
                                               body=body)
 
         assert resp.code == 200
-        assert resp.body == b'{"predictions": [["foo", 1, "pink"]]}'
-        assert resp.headers['content-type'] == "application/x-www-form-urlencoded"
+        assert resp.headers['content-type'] == "application/json"
         assert resp.headers['ce-specversion'] == "1.0"
-        assert resp.headers['ce-id'] == "36077800-0c23-4f38-a0b4-01f4369f670a"
-        assert resp.headers['ce-source'] == "https://example.com/event-producer"
-        assert resp.headers['ce-type'] == "com.example.sampletype1"
-        assert resp.headers['ce-datacontenttype'] == "application/x-www-form-urlencoded"
+        assert resp.headers["ce-id"] != "36077800-0c23-4f38-a0b4-01f4369f670a"
+        assert resp.headers['ce-source'] == "io.kserve.kfserver.TestModel"
+        assert resp.headers['ce-type'] == "io.kserve.inference.response"
         assert resp.headers['ce-time'] > "2021-01-28T21:04:43.144141+00:00"
+        assert resp.body == b'{"predictions": [["foo", 1, "pink"]]}'