Repository: kubeflow/manifests
Pull request #3056: Document secure Kserve authentication via automated tests
Author: login: madmecodes
Labels: name: lgtm
name: approved
name: size/L
Created: 2025-03-18T13:42:28Z | Merged: 2025-03-25T19:02:59Z

# Pull Request Template for Kubeflow Manifests

## ✏️ Summary of Changes
This PR addresses a security issue with KServe inference endpoints by applying oauth2-proxy authentication to the cluster-local-gateway. Currently, KServe endpoints lack proper authentication because they use the cluster-local-gateway which has an `ALLOW` policy instead of proper oauth2-proxy authentication.

Changes include:
1. Modified the cluster-local-gateway AuthorizationPolicy to use oauth2-proxy authentication instead of allowing all traffic without authentication
2. Updated the KServe test workflow to:
   - Use proper Kubeflow user namespaces instead of generic test namespaces
   - Test with tokens from the appropriate service accounts
   - Apply predictor-specific AuthorizationPolicy for tests

This ensures KServe inference endpoints are properly secured and require authentication, just like other Kubeflow services accessed through the istio-ingressgateway.

## 📦 Dependencies
No direct dependencies on other PRs. This PR combines improvements that were planned for PR #2936 with the security fix.

## 🐛 Related Issues
- Fixes #2811 (Secure KServe inferencing with proper authentication)
- Related to #2936 (Update KServe tests with real Kubeflow namespaces)
- Helps unblock #2907 (ISTIO CNI by default)

## ✅ Contributor Checklist
  - [x] I have tested these changes with kustomize. See [Installation Prerequisites](https://github.com/kubeflow/manifests#prerequisites).
  - [x] All commits are [_signed-off_](https://github.com/kubeflow/community/tree/master/dco-signoff-hook#signing-off-commits) to satisfy the DCO check.

---     
 
> You can join the CNCF Slack and access our meetings at the [Kubeflow Community](https://www.kubeflow.org/docs/about/community/) website. Our channel on the CNCF Slack is here [**#kubeflow-platform**](https://app.slack.com/client/T08PSQ7BQ/C073W572LA2).
  


Linked issue #2811: Enable and document for Kubeflow 1.10 Kserve secure inferencing from inside and outside the cluster with tokens
### Validation Checklist

- [X] Is this a Kubeflow issue?
- [X] Are you posting in the right repository ?
- [X] Did you follow the installation guide https://github.com/kubeflow/manifests?tab=readme-ov-file ?
- [X] Is the issue report properly structured and detailed with version numbers?
- [X] Is this for Kubeflow development ?
- [X] Would you like to work on this issue?
- [X] You can join the CNCF Slack and access our meetings at the [Kubeflow Community](https://www.kubeflow.org/docs/about/community/) website. Our channel on the CNCF Slack is here [**#kubeflow-platform**](https://app.slack.com/client/T08PSQ7BQ/C073W572LA2).

### Version

master

### Describe your issue

From @kromanow94 

I did some investigation and I found out that this is because the `VirtualServices` created by `kserve` are configured by default to use `cluster-local-gateway`. `istio-ingressgateway` is configured with `AuthorizationPolicy` `istio-ingressgateway-oauth2-proxy` which enforces the traffic to go through `oauth2-proxy`. There is no such `AuthorizationPolicy` for `cluster-local-gateway`.

So, I see two options:
1. Configure Istio auth for current setup with `cluster-local-gateway`
    1. Create `cluster-local-gateway-oauth2-proxy` `AuthorizationPolicy` to enforce authentication with `oauth2-proxy`:
        ```yaml
        apiVersion: security.istio.io/v1
        kind: AuthorizationPolicy
        metadata:
          name: cluster-local-gateway-oauth2-proxy
          namespace: istio-system
        spec:
          action: CUSTOM
          provider:
            name: oauth2-proxy
          rules:
          - {}
          selector:
            matchLabels:
              app: cluster-local-gateway
        ```
    2. Depending on your setup, if the model is deployed in Kubeflow managed namespace (KF Profile, for example `kubeflow-user-example-com`), you also have to configure access to the `sklearn-iris` deployment:
        ```yaml
        apiVersion: security.istio.io/v1beta1
        kind: AuthorizationPolicy
        metadata:
          name: sklearn-iris-predictor-allow
          namespace: kubeflow-user-example-com
        spec:
          selector:
            matchLabels:
              serving.knative.dev/service: sklearn-iris-predictor
          action: ALLOW
          rules:
          - {}
        ```
    3. Testing with `curl`:
        ```bash
        $ curl -XPOST -v "http://sklearn-iris.kubeflow-user-example-com.svc.cluster.local/v1/models/sklearn-iris:predict" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -d '{"instances": [[6.8,  2.8,  4.8,  1.4], [6.0,  3.4,  4.5,  1.6]]}' -H "Content-Type: application/json"
        Note: Unnecessary use of -X or --request, POST is already inferred.
        * Host sklearn-iris.kubeflow-user-example-com.svc.cluster.local:80 was resolved.
        * IPv6: (none)
        * IPv4: 172.20.1.23
        *   Trying 172.20.1.23:80...
        * Connected to sklearn-iris.kubeflow-user-example-com.svc.cluster.local (172.20.1.23) port 80
        > POST /v1/models/sklearn-iris:predict HTTP/1.1
        > Host: sklearn-iris.kubeflow-user-example-com.svc.cluster.local
        > User-Agent: curl/8.7.1
        > Accept: */*
        > Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ikh3ZUQ2enNYYnRZNUFZQk8xX1ZKc3ZCZGwwRmR3dTdwRURiQXpDN3c5MncifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdCJdLCJleHAiOjIwMjg1NDA4NzAsImlhdCI6MTcxMzE4MDg3MCwiaXNzIjoiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6Imt1YmVmbG93LXVzZXItZXhhbXBsZS1jb20iLCJwb2QiOnsibmFtZSI6ImN1cmwiLCJ1aWQiOiI3ZGI2ZjliNC0zZTliLTQ3ZDUtOWI4ZC0yZjhiMWVkNTVhZjkifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQtZWRpdG9yIiwidWlkIjoiODZhZmM3OGYtMTIzYS00MDMwLWI5YjQtZDllYWQ5YmE2NTc4In19LCJuYmYiOjE3MTMxODA4NzAsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlZmxvdy11c2VyLWV4YW1wbGUtY29tOmRlZmF1bHQtZWRpdG9yIn0.iY9WY7vqFQvxv3mzFYlnKQ3arG631movAfkIM1eWH_UdsQuWUupIz7wak81pOM23gBPpYxMT5HR1ZgVHYWG07Neh4e1ySUzhmPNNfydSIs-jUP1P8BjEPq3BdSQ9j_1pGggMDXFM4msFnEdAjlmpl23yDKOoJCj0RDV3fZIiA-mf7wLyiv_E38ah1ygZXYjrTCdzstCH02aZ7VCLc1dPETttE7nlF3YoaurwHJzZF6WHXmQlVdU2yMg0RT8uRDBUDI6WTq_guxjuEBEJrj166pXbp1MBvslMBUYXPV3StQ-AXnvQUyCBDoa5NOlJKOht3UOhGeS_-1A50ctjsl8xKw
        > Content-Type: application/json
        > Content-Length: 65
        > 
        * upload completely sent off: 65 bytes
        < HTTP/1.1 200 OK
        < content-length: 21
        < content-type: application/json
        < date: Mon, 15 Apr 2024 12:47:28 GMT
        < server: envoy
        < x-envoy-upstream-service-time: 9
        < 
        * Connection #0 to host sklearn-iris.kubeflow-user-example-com.svc.cluster.local left intact
        {"predictions":[1,1]}
        ```
2. Change the kserve config to use `istio-ingressgateway` instead of `cluster-local-gateway`. This touches kserve which I don't have a lot of experience with. I tried changing the `inferenceservice-config` `ConfigMap` to define the `"localGatewayService": "istio-ingressgateway.istio-system.svc.cluster.local"` and `"localGateway": "kubeflow/kubeflow-gateway"` but that didn't work for some reason, probably something is missing...

@juliusvonkohout do you think we should add this `AuthorizationPolicy` for `cluster-local-gateway` to the manifests?

### Steps to reproduce the issue

See above

### Put here any screenshots or videos (optional)

_No response_
Linked issue comments: user: login: juliusvonkohout
created_at: 2024-07-22T17:52:38Z
body: @kromanow94 yes we should tackle this for 1.10
author_association: MEMBER
user: login: kromanow94
created_at: 2024-07-23T05:49:03Z
body: @juliusvonkohout , makes sense. From conceptual point of view, what do we expect and what are the assumptions?

I guess it's easy decision for Out Of Cluster - the `Authorization` header should contain a valid JWT.

What about from in-cluster? Do we expect that access to given kserve endpoint should be enabled without `Authorization` header when called from the same namespace? From other namespace we probably want to use a valid JWT either way so it's similar as from out of cluster, just the istio ingress gateway would be `cluster-local`.
author_association: CONTRIBUTOR
user: login: juliusvonkohout
created_at: 2024-07-23T14:31:07Z
body: "Do we expect that access to given kserve endpoint should be enabled without Authorization header when called from the same namespace?" I am not sure. Actually i like to enforce security everywhere. @kimwnasptd what do you think?
author_association: MEMBER
user: login: kromanow94
created_at: 2024-07-24T09:32:38Z
body: > Actually i like to enforce security everywhere

That would also ease and unify the approach from implementation perspective.
author_association: CONTRIBUTOR
user: login: padrian2s
created_at: 2024-08-07T20:29:41Z
body: I bump into approval_prompt for v1.9, but yes the docs are missing

curl -XPOST -v "http://sklearn-iris.kubeflow-user-example-com.svc.cluster.local/v1/models/sklearn-iris:predict" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -d '{"instances": [[6.8,  2.8,  4.8,  1.4], [6.0,  3.4,  4.5,  1.6]]}' -H "Content-Type: application/json"
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 172.20.112.202:80...
* Connected to sklearn-iris.kubeflow-user-example-com.svc.cluster.local (172.20.112.202) port 80 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kubeflow-user-example-com.svc.cluster.local
> User-Agent: curl/7.88.1
> Accept: */*
> Authorization: Bearer <<REMOVED>>
> Content-Type: application/json
> Content-Length: 65
> 
< HTTP/1.1 302 Found
< location: /dex/auth?approval_prompt=force&client_id=kubeflow-oidc-authservice&code_challenge=xzijon7LEPKr6gr3FsDcXOYcw9dSVgpN0aof3dyehSo&code_challenge_method=S256&redirect_uri=%2Foauth2%2Fcallback&response_type=code&scope=profile+email+groups+openid&state=YwJa9xq0i_wZuyC_RE8P90Fgb5eybkRasgrk6q5Nd0o%3A%2Fv1%2Fmodels%2Fsklearn-iris%3Apredict
< set-cookie: oauth2_proxy_kubeflow_csrf=avnayyw40z0m5rKMuBfHOjugZwMS94CxZTj04EJ_SVtoCAM5EixXNlzgRLj9Qf2V0Y3ARRgtBRnZGkqQQR7DVQaZuAUBENgnfPoxQD3wrWS6c3r5GwZaFhryWgssC39nnsN92984toftiQDWenP1NtqY6UQUMd4mt6gjQb3rWB62Rd3hwv6l1p36HAOHwFi-FhFBwPHgsgA7hcXiULacdx9ycmjkWoJZHFchLUGPleVpjkdTxQXfVgWDF7Gh5A==|1723062375|VMndKKL-bTECRokaEjTfvrEAPhfoZfGLsIJmf9BvhYI=; Path=/; Expires=Wed, 07 Aug 2024 20:41:15 GMT; HttpOnly
< date: Wed, 07 Aug 2024 20:26:15 GMT
< server: envoy
< x-envoy-upstream-service-time: 1
< content-length: 0
< 
* Connection #0 to host sklearn-iris.kubeflow-user-example-com.svc.cluster.local left intact
author_association: NONE
user: login: juliusvonkohout
created_at: 2024-08-08T06:52:06Z
body: "@juliusvonkohout do you think we should add this AuthorizationPolicy for cluster-local-gateway to the manifests?" Yes, we have to move one. This is too complex for many users to understand and we need to provide it out of the box.
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2024-08-08T06:56:02Z
body: So @kromanow94  please create a PR if you have time.
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2024-08-15T06:25:11Z
body: There seems to be a lot of wrong/outdated stuff in https://github.com/kserve/kserve/pull/3260 and we should probably fix it there as well.
author_association: MEMBER
user: login: jaffe-fly
created_at: 2024-08-15T09:10:56Z
body: by my test in kubeflow 1.9 cluster, https://github.com/kserve/kserve/tree/master/docs/samples/istio-dex,this is cant run .
`_auth_session["authservice_session"]` is None.my `_auth_session` is
```
{'endpoint_url': 'http://xxxxxxx:18080/', 'redirect_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'dex_login_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'is_secured': True, 'session_cookie': 'oauth2_proxy_kubeflow_csrf=-GOrJQeg-v5_Dc_3HD12Td80HAuE9ld2eT3Y7dT2lyw1b2wUKEZaYJVjhqaBF7UW_elSQ8i08WsjXTGO2kys6VTPJWcHQ2uI_qMFqFp7Wn2_SIpjihvsfa-JX8VC-8Qag25f0jkbi-LMSu0IlRjVYoQZ2VDkriRa3bCq8j3zVOboKjcxVGujyrggNU1X4r-PyYVOujmdv7oxhw1wcS2HbYiQ84GZxPbb0S9FeFI8u2gkSvT5U_HQrpfl2MkoBg==|1723645690|pZzKszJaoId5Vb5vQN0-a99zY29WdQ2TfHtHdsIga-U=', 'authservice_session': None}
```
this issue will  solve the question?
author_association: NONE
user: login: juliusvonkohout
created_at: 2024-08-15T09:58:26Z
body: > by my test in kubeflow 1.9 cluster, https://github.com/kserve/kserve/tree/master/docs/samples/istio-dex,this is cant run . `_auth_session["authservice_session"]` is None.my `_auth_session` is
> 
> ```
> {'endpoint_url': 'http://xxxxxxx:18080/', 'redirect_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'dex_login_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'is_secured': True, 'session_cookie': 'oauth2_proxy_kubeflow_csrf=-GOrJQeg-v5_Dc_3HD12Td80HAuE9ld2eT3Y7dT2lyw1b2wUKEZaYJVjhqaBF7UW_elSQ8i08WsjXTGO2kys6VTPJWcHQ2uI_qMFqFp7Wn2_SIpjihvsfa-JX8VC-8Qag25f0jkbi-LMSu0IlRjVYoQZ2VDkriRa3bCq8j3zVOboKjcxVGujyrggNU1X4r-PyYVOujmdv7oxhw1wcS2HbYiQ84GZxPbb0S9FeFI8u2gkSvT5U_HQrpfl2MkoBg==|1723645690|pZzKszJaoId5Vb5vQN0-a99zY29WdQ2TfHtHdsIga-U=', 'authservice_session': None}
> ```
> 
> this issue will solve the question?

Hello, as said before, the kserve documentation is wrong there. Check the first post here on how to get it done with oauth2-proxy.
author_association: MEMBER
user: login: padrian2s
created_at: 2024-08-15T14:37:46Z
body: KServe is right but for internal K8s JWT tokens not with OAuth-proxy tokens that are generated outside the cluster.
author_association: NONE
user: login: juliusvonkohout
created_at: 2024-08-16T12:54:29Z
body: No, oauth2-proxy is explicitly there to use serviceaccountokens. Whether from inside or outside does not matter.
author_association: MEMBER
user: login: github-actions[bot]
created_at: 2024-10-16T00:19:48Z
body: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
author_association: NONE
user: login: juliusvonkohout
created_at: 2024-10-16T06:51:12Z
body: /lifecycle frozen
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2024-11-04T08:35:17Z
body: See also 
https://github.com/kubeflow/dashboard/issues/13 and
https://github.com/kubeflow/kubeflow/pull/6627
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2024-12-18T09:02:43Z
body: Implementation started in https://github.com/kubeflow/manifests/pull/2936
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-02-25T07:41:14Z
body: @KataaOmni do you want to help with https://github.com/kubeflow/manifests/pull/2936 ?
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-03-25T19:03:35Z
body: See https://github.com/kubeflow/manifests/pull/3056#issuecomment-2752093237 for the reopen
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-03-25T20:27:56Z
body:  In the follow up PR you can either remove or secure the cluster-local-gateway. Whatever works for kserve and keeps all  security tests green.
author_association: MEMBER
user: login: nithin8702
created_at: 2025-04-12T16:45:45Z
body: Hi, Will this work in https protocol?
author_association: NONE
user: login: juliusvonkohout
created_at: 2025-04-12T20:23:12Z
body: > Hi, Will this work in https protocol?

Https ends at the istio ingressgateway. That should be enough for most users. With istio you then get mTLS.
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-06-13T11:07:46Z
body: Having merged a few PRs we still need to refine the gateways and authorizationpolicies for improved security.
author_association: MEMBER

Diff:
diff --git a/.github/workflows/centraldashboard_test.yaml b/.github/workflows/centraldashboard_test.yaml
index a3cc0313e7..58aade1f77 100644
--- a/.github/workflows/centraldashboard_test.yaml
+++ b/.github/workflows/centraldashboard_test.yaml
@@ -5,7 +5,7 @@ on:
     - tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh
     - .github/workflows/centraldashboard_test.yaml
     - apps/centraldashboard/upstream/**
-    - tests/gh-actions/install_istio.sh
+    - tests/gh-actions/install_istio*.sh
     - common/istio*/**
 
 jobs:
@@ -21,9 +21,10 @@ jobs:
     - name: Install Istio
       run: ./tests/gh-actions/install_istio.sh
 
-    - name: Build & Apply manifests
+    - name: Create kubeflow namespace
+      run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
+
+    - name: Install central-dashboard
       run: |
-        cd apps/centraldashboard/upstream
-        kubectl create ns kubeflow
-        kustomize build overlays/kserve | kubectl apply -f -
+        kustomize build apps/centraldashboard/upstream/overlays/kserve | kubectl apply -f -
         kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout 180s
diff --git a/.github/workflows/dex_oauth2-proxy_test.yaml b/.github/workflows/dex_oauth2-proxy_test.yaml
index 721d67fbfd..d8dc0b2432 100644
--- a/.github/workflows/dex_oauth2-proxy_test.yaml
+++ b/.github/workflows/dex_oauth2-proxy_test.yaml
@@ -9,7 +9,7 @@ on:
     - common/istio*/**
     - experimental/security/PSS/*
     - common/dex/base/**
-    - tests/gh-actions/install_istio.sh
+    - tests/gh-actions/install_istio*.sh
 
 jobs:
   build:
@@ -47,11 +47,24 @@ jobs:
         echo "Waiting for pods in auth namespace to become ready..."
         kubectl wait --for=condition=ready pods --all --timeout=180s -n auth
 
-    - name: Build & Apply manifests
+    - name: Install central-dashboard
       run: |
-        while ! kustomize build ./tests/gh-actions/deploy-dex-login-environment | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 20; done
+        kustomize build apps/centraldashboard/upstream/overlays/kserve | kubectl apply -f -
         kubectl wait --for=condition=Ready pods --all --all-namespaces --timeout 180s
 
+    - name: Create KF Profile
+      run: |
+        kustomize build common/user-namespace/base | kubectl apply -f -
+        sleep 30 # for the Profile controller to create the namespace from the profile
+        PROFILE_CONTROLLER_POD=$(kubectl get pods -n kubeflow -o json | jq -r '.items[] | select(.metadata.name | startswith("profiles-deployment")) | .metadata.name')
+        if [[ -z "$PROFILE_CONTROLLER_POD" ]]; then
+          echo "Error: profiles-deployment pod not found in kubeflow namespace."
+          exit 1
+        fi
+        kubectl logs -n kubeflow "$PROFILE_CONTROLLER_POD"
+        KF_PROFILE=kubeflow-user-example-com
+        kubectl -n $KF_PROFILE get pods,configmaps,secrets
+
     - name: port forward
       run: |
         ingress_gateway_service=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}')
diff --git a/.github/workflows/kserve_m2m_test.yaml b/.github/workflows/kserve_m2m_test.yaml
index a7b1aa5269..87b275cb4e 100644
--- a/.github/workflows/kserve_m2m_test.yaml
+++ b/.github/workflows/kserve_m2m_test.yaml
@@ -7,7 +7,7 @@ on:
     - apps/kserve/**
     - tests/gh-actions/install_kserve.sh
     - common/istio*/**
-    - tests/gh-actions/install_istio.sh
+    - tests/gh-actions/install_istio*.sh
     - common/oauth2-proxy/**
     - tests/gh-actions/install_oauth2-proxy.sh
     - common/cert-manager/**
@@ -15,7 +15,6 @@ on:
     - common/knative/**
     - tests/gh-actions/install_knative.sh
 
-
 jobs:
   build:
     runs-on: ubuntu-latest
@@ -29,11 +28,8 @@ jobs:
     - name: Install kubectl
       run: ./tests/gh-actions/install_kubectl.sh
 
-    - name: Create kubeflow namespace
-      run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
-
-    - name: Install Istio
-      run: ./tests/gh-actions/install_istio.sh
+    - name: Install Istio CNI
+      run: ./tests/gh-actions/install_istio-cni.sh
 
     - name: Install oauth2-proxy
       run: ./tests/gh-actions/install_oauth2-proxy.sh
@@ -41,14 +37,119 @@ jobs:
     - name: Install cert-manager
       run: ./tests/gh-actions/install_cert_manager.sh
 
-    - name: Install knative
-      run: ./tests/gh-actions/install_knative.sh
+    - name: Create kubeflow namespace
+      run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -
+
+    - name: Install knative CNI
+      run: ./tests/gh-actions/install_knative-cni.sh
 
     - name: Install KServe
       run: ./tests/gh-actions/install_kserve.sh
 
-    - name: Create test namespace # TODO to be removed and instead we shall use kubeflow-user-example-com
-      run: kubectl create ns kserve-test
+    - name: Install KF Multi Tenancy
+      run: ./tests/gh-actions/install_multi_tenancy.sh
+
+    - name: Install kubeflow-istio-resources
+      run: kustomize build common/istio-1-24/kubeflow-istio-resources/base | kubectl apply -f -
+
+    - name: Create KF Profile
+      run: |
+        kustomize build common/user-namespace/base | kubectl apply -f -
+        sleep 30 # for the Profile controller to create the namespace from the profile
+
+        PROFILE_CONTROLLER_POD=$(kubectl get pods -n kubeflow -o json | jq -r '.items[] | select(.metadata.name | startswith("profiles-deployment")) | .metadata.name')
+        if [[ -z "$PROFILE_CONTROLLER_POD" ]]; then
+          echo "Error: profiles-deployment pod not found in kubeflow namespace."
+          exit 1
+        fi
+        kubectl logs -n kubeflow "$PROFILE_CONTROLLER_POD"
+        KF_PROFILE=kubeflow-user-example-com
+        kubectl -n $KF_PROFILE get pods,configmaps,secrets
+
+    - name: Diagnose KServe Service Labels
+      run: |
+        echo "=== KServe Predictor Service Labels ==="
+        kubectl get pods -n kubeflow-user-example-com -l serving.knative.dev/service=isvc-sklearn-predictor-default --show-labels
+
+    # TODO for follow up PR
+    #- name: Apply KServe predictor AuthorizationPolicy
+    #  run: |
+    #      cat <<EOF | kubectl apply -f -
+    #      apiVersion: security.istio.io/v1beta1
+    #      kind: AuthorizationPolicy
+    #      metadata:
+    #        name: sklearn-iris-predictor-allow
+    #        namespace: kubeflow-user-example-com
+    #      spec:
+    #        selector:
+    #          matchLabels:
+    #            serving.knative.dev/service: isvc-sklearn-predictor
+    #        action: ALLOW
+    #        rules:
+    #        - from:
+    #            - source:
+    #                namespaces:
+    #                - "istio-system"
+    #                - "knative-serving"
+    #               - "kubeflow"
+    #              - "kubeflow-user-example-com"
+    #            - principals:
+    #              - "cluster.local/ns/kubeflow-user-example-com/sa/default-editor"
+    #              - "cluster.local/ns/kubeflow-user-example-com/sa/default"
+    #              - "cluster.local/ns/kubeflow-user-example-com/sa/default-viewer"
+    #          to:
+    #            - operation:
+    #                paths:
+    #                - "/v1/models/*"
+    #                - "/v2/models/*"
+    #     EOF
+
+    - name: Apply INSECURE KServe AuthorizationPolicy
+      run: |
+          cat <<EOF | kubectl apply -f -
+          apiVersion: security.istio.io/v1beta1
+          kind: AuthorizationPolicy
+          metadata:
+            name: allow-in-cluster-kserve
+            namespace: kubeflow-user-example-com
+          spec:
+            rules:
+              - to:
+                  - operation:
+                      paths:
+                        - /v1/models/*
+                        - /v2/models/*
+          EOF
+
+    - name: Add KServe path-based routing for external access
+      run: |
+        cat <<EOF | kubectl apply -f -
+        apiVersion: networking.istio.io/v1beta1
+        kind: VirtualService
+        metadata:
+          name: isvc-sklearn-external
+          namespace: kubeflow-user-example-com
+        spec:
+          gateways:
+            - kubeflow/kubeflow-gateway
+          hosts:
+            - '*'
+          http:
+            - match:
+                - uri:
+                    prefix: /kserve/kubeflow-user-example-com/isvc-sklearn/
+              rewrite:
+                uri: /
+              route:
+                - destination:
+                    host: knative-local-gateway.istio-system.svc.cluster.local
+                  headers:
+                    request:
+                      set:
+                        Host: isvc-sklearn-predictor-default.kubeflow-user-example-com.svc.cluster.local
+                  weight: 100
+              timeout: 300s
+        EOF
 
     - name: Setup python 3.12
       uses: actions/setup-python@v4
@@ -64,26 +165,97 @@ jobs:
         nohup kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80 &
         while ! curl localhost:8080; do echo waiting for port-forwarding; sleep 1; done; echo port-forwarding ready
 
-    - name: Run kserve tests with m2m token from SA default/default # TODO Run kserve tests with m2m token from SA kubeflow-user-example-com/default-editor
+    - name: Verify CNI is being used instead of init containers
+      run: |
+        INIT_CONTAINERS=$(kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.initContainers[*]}{.name}{"\n"}{end}{"\n"}{end}' | grep istio-init || echo "none")
+        if [[ "$INIT_CONTAINERS" != "none" ]]; then
+          echo "Error: Found istio-init containers, CNI is not being used properly"
+          echo "$INIT_CONTAINERS"
+          exit 1
+        else
+          echo "Success: No istio-init containers found, CNI is working correctly"
+        fi
+
+    - name: Run kserve tests with m2m token from SA kubeflow-user-example-com/default-editor
       run: |
-        # TODO run the tests against  kubeflow-user-example-com
         export KSERVE_INGRESS_HOST_PORT=localhost:8080
-        export KSERVE_M2M_TOKEN="$(kubectl -n default create token default)"
-        # TODO export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
-        # TODO in contrib/kserve/tests/utils.py use KSERVE_TEST_NAMESPACE = "kubeflow-user-example-com"
+        export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
         cd ./apps/kserve/tests && pytest . -vs --log-level info
 
-    - name: Run and fail kserve tests without kserve m2m token
+    - name: Detailed KServe Access Diagnostics
       run: |
         export KSERVE_INGRESS_HOST_PORT=localhost:8080
-        cd ./apps/kserve/tests
-        if pytest . -vs --log-level info; then
-          echo "This test should fail with an HTTP redirect to oauth2-proxy/dex auth."; exit 1
-        else
-          echo "Task failed successfully!"
-          echo "This is a provisional way of testing that m2m is enabled for kserve."
-        fi
+        export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
+
+        echo "=== AuthorizationPolicy Details ==="
+        kubectl get authorizationpolicy -n kubeflow-user-example-com -o yaml
+
+        echo "=== Detailed Curl Test ==="
+        curl -vv \
+          -H "Host: isvc-sklearn.kubeflow-user-example-com.example.com" \
+          -H "Authorization: Bearer ${KSERVE_M2M_TOKEN}" \
+          -H "Content-Type: application/json" \
+          "http://${KSERVE_INGRESS_HOST_PORT}/v1/models/isvc-sklearn:predict" \
+          -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}'
+
+    # TODO FOR FOLLOW UP PR
+    #- name: Run and fail kserve tests without kserve m2m token
+      #run: |
+      #  export KSERVE_INGRESS_HOST_PORT=localhost:8080
+      #  cd ./apps/kserve/tests
+      #  if pytest . -vs --log-level info; then
+      #    echo "This test should fail with an HTTP redirect to oauth2-proxy/dex auth."; exit 1
+      #  else
+      #    echo "Task failed successfully!"
+      #    echo "This is a provisional way of testing that m2m is enabled for kserve."
+      #  fi
+
+    # TODO FOR FOLLOW UP PR
+    #- name: Test that token from attacker namespace is rejected
+    #  run: |
+    #    export KSERVE_INGRESS_HOST_PORT=localhost:8080
+    #    kubectl create ns kubeflow-user-example-com-attacker
+    #    kubectl create serviceaccount attacker-sa -n kubeflow-user-example-com-attacker
+    #    export ATTACKER_TOKEN="$(kubectl -n kubeflow-user-example-com-attacker create token attacker-sa)"
+    #    RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" -H "Host: isvc-sklearn.kubeflow-user-example-com.example.com" \
+    #        -H "Authorization: Bearer ${ATTACKER_TOKEN}" \
+    #        -H "Content-Type: application/json" \
+    #        "http://${KSERVE_INGRESS_HOST_PORT}/v1/models/isvc-sklearn:predict" \
+    #        -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}')
+    #    if [[ "$RESPONSE" == "403" || "$RESPONSE" == "401" ]]; then
+    #      echo "Security test passed: Request with attacker token was correctly rejected with $RESPONSE"
+    #    else
+    #      echo "Security test failed: Request with attacker token returned $RESPONSE instead of 403/401"
+    #      exit 1
+    #    fi
+
+    - name: Test path-based external access
+      run: |
+        export KSERVE_INGRESS_HOST_PORT=localhost:8080
+        export KSERVE_M2M_TOKEN="$(kubectl -n kubeflow-user-example-com create token default-editor)"
+
+        # Test external path-based access
+        curl -v -H "Host: isvc-sklearn.kubeflow-user-example-com.example.com" \
+            -H "Authorization: Bearer ${KSERVE_M2M_TOKEN}" \
+            -H "Content-Type: application/json" \
+            "http://${KSERVE_INGRESS_HOST_PORT}/kserve/kubeflow-user-example-com/isvc-sklearn/v1/models/isvc-sklearn:predict" \
+            -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}'
 
     - name: Run kserve models webapp test
       run: |
         kubectl wait --for=condition=Available --timeout=300s -n kubeflow deployment/kserve-models-web-app
+
+    - name: Apply Pod Security Standards baseline levels
+      run: ./tests/gh-actions/enable_baseline_PSS.sh
+
+    - name: Unapply applied baseline labels
+      run: |
+        NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow" "knative-serving")
+        for NAMESPACE in "${NAMESPACES[@]}"; do
+          if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then
+            kubectl label namespace $NAMESPACE pod-security.kubernetes.io/enforce-
+          fi
+        done
+
+    - name: Applying Pod Security Standards restricted levels
+      run: ./tests/gh-actions/enable_restricted_PSS.sh
diff --git a/apps/kserve/tests/requirements.txt b/apps/kserve/tests/requirements.txt
index ac17f9f373..27c9f1fef8 100644
--- a/apps/kserve/tests/requirements.txt
+++ b/apps/kserve/tests/requirements.txt
@@ -1,4 +1,4 @@
 pytest>=7.0.0
-kserve>=0.12.1
+kserve>=0.14.1
 kubernetes>=18.20.0
 requests>=2.18.4
diff --git a/apps/kserve/tests/utils.py b/apps/kserve/tests/utils.py
index 77b733b6eb..f3ad0e7d91 100644
--- a/apps/kserve/tests/utils.py
+++ b/apps/kserve/tests/utils.py
@@ -26,7 +26,7 @@
 logging.basicConfig(level=logging.INFO)
 
 KSERVE_NAMESPACE = "kserve"
-KSERVE_TEST_NAMESPACE = "kserve-test"
+KSERVE_TEST_NAMESPACE = "kubeflow-user-example-com"
 MODEL_CLASS_NAME = "modelClass"
 
 
diff --git a/common/istio-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml b/common/istio-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
index 4a45b0a1e0..42db13d36e 100644
--- a/common/istio-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
+++ b/common/istio-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
@@ -1,14 +1,16 @@
-# Allow all traffic to the cluster-local-gateway
+# Enforce OAuth2-proxy authentication for cluster-local-gateway
 apiVersion: security.istio.io/v1beta1
 kind: AuthorizationPolicy
 metadata:
-  name: cluster-local-gateway
+  name: cluster-local-gateway-oauth2-proxy
 spec:
-  action: ALLOW
+  action: CUSTOM
+  provider:
+    name: oauth2-proxy
   selector:
     # Same as the cluster-local-gateway Service selector
     matchLabels:
       app: cluster-local-gateway
       istio: cluster-local-gateway
   rules:
-  - {}
\ No newline at end of file
+  - {}
diff --git a/common/istio-cni-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml b/common/istio-cni-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
index 4a45b0a1e0..42db13d36e 100644
--- a/common/istio-cni-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
+++ b/common/istio-cni-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
@@ -1,14 +1,16 @@
-# Allow all traffic to the cluster-local-gateway
+# Enforce OAuth2-proxy authentication for cluster-local-gateway
 apiVersion: security.istio.io/v1beta1
 kind: AuthorizationPolicy
 metadata:
-  name: cluster-local-gateway
+  name: cluster-local-gateway-oauth2-proxy
 spec:
-  action: ALLOW
+  action: CUSTOM
+  provider:
+    name: oauth2-proxy
   selector:
     # Same as the cluster-local-gateway Service selector
     matchLabels:
       app: cluster-local-gateway
       istio: cluster-local-gateway
   rules:
-  - {}
\ No newline at end of file
+  - {}
diff --git a/tests/gh-actions/deploy-dex-login-environment/kustomization.yaml b/tests/gh-actions/deploy-dex-login-environment/kustomization.yaml
deleted file mode 100644
index c4c59cf063..0000000000
--- a/tests/gh-actions/deploy-dex-login-environment/kustomization.yaml
+++ /dev/null
@@ -1,58 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-
-sortOptions:
-  order: legacy
-  legacySortOptions:
-    orderFirst:
-    - Namespace
-    - ResourceQuota
-    - StorageClass
-    - CustomResourceDefinition
-    - MutatingWebhookConfiguration
-    - ServiceAccount
-    - PodSecurityPolicy
-    - NetworkPolicy
-    - Role
-    - ClusterRole
-    - RoleBinding
-    - ClusterRoleBinding
-    - ConfigMap
-    - Secret
-    - Endpoints
-    - Service
-    - LimitRange
-    - PriorityClass
-    - PersistentVolume
-    - PersistentVolumeClaim
-    - Deployment
-    - StatefulSet
-    - CronJob
-    - PodDisruptionBudget
-    orderLast:
-    - ValidatingWebhookConfiguration
-
-resources:
-# Istio
-- ../../../common/istio-1-24/istio-crds/base
-- ../../../common/istio-1-24/istio-namespace/base
-- ../../../common/istio-1-24/istio-install/overlays/oauth2-proxy
-# oauth2-proxy
-- ../../../common/oauth2-proxy/overlays/m2m-dex-and-kind
-# Dex
-- ../../../common/dex/overlays/oauth2-proxy
-- ../../../common/istio-1-24/cluster-local-gateway/base
-# Kubeflow namespace
-- ../../../common/kubeflow-namespace/base
-# NetworkPolicies
-- ../../../common/networkpolicies/base
-# Kubeflow Roles
-- ../../../common/kubeflow-roles/base
-# Kubeflow Istio Resources
-- ../../../common/istio-1-24/kubeflow-istio-resources/base
-# Central Dashboard
-- ../../../apps/centraldashboard/overlays/oauth2-proxy
-# Profiles + KFAM
-- ../../../apps/profiles/upstream/overlays/kubeflow
-# User namespace
-- ../../../common/user-namespace/base


Reviews:
user: login: juliusvonkohout
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T12:19:04Z
user: login: juliusvonkohout
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T12:19:34Z
user: login: juliusvonkohout
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T12:23:49Z
user: login: juliusvonkohout
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T12:25:27Z
user: login: juliusvonkohout
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T12:27:09Z
user: login: juliusvonkohout
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-23T13:21:18Z
user: login: akagami-harsh
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-23T23:31:05Z

Review comments:
path: .github/workflows/kserve_cni_test.yaml
user: login: juliusvonkohout
body: this file will be deleted if you rebase
created_at: 2025-03-20T12:19:03Z
author_association: MEMBER
path: .github/workflows/kserve_test.yaml
user: login: juliusvonkohout
body: this file will be deleted when you rebase to master
created_at: 2025-03-20T12:19:34Z
author_association: MEMBER
path: .github/workflows/kserve_m2m_test.yaml
user: login: juliusvonkohout
body:     - name: Create kubeflow namespace
      run: kustomize build common/kubeflow-namespace/base | kubectl apply -f -

can be removed because it is a duplicate of 

    - name: Create KF Profile
      run: kustomize build common/user-namespace/base | kubectl apply -f -

It helps if 

    - name: Create KF Profile
      run: kustomize build common/user-namespace/base | kubectl apply -f -

comes before 

    - name: Install KF Multi Tenancy
      run: ./tests/gh-actions/install_multi_tenancy.sh

as in https://github.com/kubeflow/manifests/blob/88801f6ce84ac7eab3e2883dd3d80cb63ccb6ab3/.github/workflows/pipeline_run_from_notebook.yaml#L45C1-L46C55
created_at: 2025-03-20T12:23:49Z
author_association: MEMBER
diff_hunk: @@ -44,11 +43,46 @@ jobs:
     - name: Install knative
       run: ./tests/gh-actions/install_knative.sh
 
-    - name: Build & Apply manifests
+    - name: Install KServe
       run: ./tests/gh-actions/install_kserve.sh
 
-    - name: Create test namespace
-      run: kubectl create ns kserve-test
+    - name: Install KF Multi Tenancy
+      run: ./tests/gh-actions/install_multi_tenancy.sh
+
+    - name: Install kubeflow-istio-resources
+      run: kustomize build common/istio-1-24/kubeflow-istio-resources/base | kubectl apply -f -
path: .github/workflows/kserve_m2m_test.yaml
user: login: juliusvonkohout
body: also this

```
    - name: Install kubeflow-istio-resources
      run: kustomize build common/istio-1-24/kubeflow-istio-resources/base | kubectl apply -f -
```
should be higher up as in https://github.com/kubeflow/manifests/blob/88801f6ce84ac7eab3e2883dd3d80cb63ccb6ab3/.github/workflows/pipeline_run_from_notebook.yaml#L39
created_at: 2025-03-20T12:25:27Z
author_association: MEMBER
diff_hunk: @@ -44,11 +43,46 @@ jobs:
     - name: Install knative
       run: ./tests/gh-actions/install_knative.sh
 
-    - name: Build & Apply manifests
+    - name: Install KServe
       run: ./tests/gh-actions/install_kserve.sh
 
-    - name: Create test namespace
-      run: kubectl create ns kserve-test
+    - name: Install KF Multi Tenancy
+      run: ./tests/gh-actions/install_multi_tenancy.sh
+
+    - name: Install kubeflow-istio-resources
+      run: kustomize build common/istio-1-24/kubeflow-istio-resources/base | kubectl apply -f -
+
+    - name: Create KF Profile
+      run: kustomize build common/user-namespace/base | kubectl apply -f -
+
+    - name: Wait for KF Profile to be ready
path: .github/workflows/kserve_m2m_test.yaml
user: login: juliusvonkohout
body: this can be removed if you rebase and address the comments above
created_at: 2025-03-20T12:27:09Z
author_association: MEMBER
path: common/istio-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
user: login: juliusvonkohout
body: @akagami-harsh can you cross check whether this file is auto generated?


created_at: 2025-03-23T13:21:18Z
author_association: MEMBER
path: common/istio-1-24/cluster-local-gateway/base/gateway-authorizationpolicy.yaml
user: login: akagami-harsh
body: The `gateway-authorizationpolicy.yaml` file appears to be a manually created file that doesn't change when running the `synchronize-istio-manifests.sh` script.
created_at: 2025-03-23T23:31:05Z
author_association: MEMBER

PR comments:
user: login: madmecodes
created_at: 2025-03-18T13:43:14Z
body: PS: okay i will look into the failed CI/CD runs, fixing them
author_association: CONTRIBUTOR
user: login: juliusvonkohout
created_at: 2025-03-18T17:22:00Z
body: /retest
author_association: MEMBER
user: login: madmecodes
created_at: 2025-03-18T17:35:48Z
body: > /retest
<img width="714" alt="Screenshot 2025-03-18 at 11 05 09 PM" src="https://github.com/user-attachments/assets/0af9ef9d-41a4-4834-9545-a4ce256194e8" />


namespace creation is failing, i am debugging that, i will tag u once the issue is solved, sorry for the inconvenience :(
author_association: CONTRIBUTOR
user: login: juliusvonkohout
created_at: 2025-03-20T12:13:52Z
body: https://github.com/kubeflow/manifests/pull/3062 is merged, so you can just rebase to master and have way less complexity
author_association: MEMBER
user: login: madmecodes
created_at: 2025-03-21T09:05:37Z
body: **if KF-Profile is before KF-multi-tenancy**
created a dependency issue.

**common/kubeflow-namespace/base  creates** kubeflow namespace
**common/user-namespace/base creates** kubeflow-user-example-com namespace and profile controllers


**If KF-profile is after  Multi Tenancy**

<img width="350" alt="Screenshot 2025-03-21 at 12 22 56 PM" src="https://github.com/user-attachments/assets/d0cc9fdd-9098-48e8-b641-d3258ef6fdfe" />

<img width="350" alt="Screenshot 2025-03-21 at 2 35 59 PM" src="https://github.com/user-attachments/assets/b62d2081-3a2a-4ce2-a015-a7053fa4a01f" />


@juliusvonkohout  Could you please review the installation order, and guide a bit, still facing the namespace issue.
author_association: CONTRIBUTOR
user: login: juliusvonkohout
created_at: 2025-03-21T13:38:50Z
body: ```
 test_sklearn.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
utils.py:67: in predict
    return predict_str(
utils.py:125: in predict_str
    response.raise_for_status()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Response [403]>

    def raise_for_status(self):
        """Raises :class:`HTTPError`, if one occurred."""
    
        http_error_msg = ""
        if isinstance(self.reason, bytes):
            # We attempt to decode utf-8 first because some servers
            # choose to localize their reason strings. If the string
            # isn't utf-8, we fall back to iso-8859-1 for all other
            # encodings. (See PR #3538)
            try:
                reason = self.reason.decode("utf-8")
            except UnicodeDecodeError:
                reason = self.reason.decode("iso-8859-1")
        else:
            reason = self.reason
    
        if 400 <= self.status_code < 500:
            http_error_msg = (
                f"{self.status_code} Client Error: {reason} for url: {self.url}"
            )
    
        elif 500 <= self.status_code < 600:
            http_error_msg = (
                f"{self.status_code} Server Error: {reason} for url: {self.url}"
            )
    
        if http_error_msg:
>           raise HTTPError(http_error_msg, response=self)
E           requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://localhost:8080/v1/models/isvc-sklearn:predict

/opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/requests/models.py:1024: HTTPError
------------------------------ Captured log call -------------------------------
INFO     root:utils.py:103 M2M Token Found.
INFO     root:utils.py:114 Sending Header = {'Host': 'isvc-sklearn.kubeflow-user-example-com.example.com', 'Content-Type': 'application/json', 'Authorization': '***'}
INFO     root:utils.py:115 Sending url = http://localhost:8080/v1/models/isvc-sklearn:predict
INFO     root:utils.py:116 Sending request data: {"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}
INFO     root:utils.py:118 Got response code 403, content b'RBAC: access denied'
=============================== warnings summary ===============================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.ScalarMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED test_sklearn.py::test_sklearn_kserve - requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://localhost:8080/v1/models/isvc-sklearn:predict
================== 1 failed, 2 warnings in 112.54s (0:01:52) ===================
Error: Process completed with exit code 1.
```
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-03-21T14:34:35Z
body: 1. Please try first what was suggested in https://github.com/kubeflow/manifests/issues/2811

2. Then you can also try 
``` 
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-in-cluster-kserve
spec:
  rules:
    - to:
        - operation:
            paths:
              - /v1/models/*
              - /v2/models/*
```

which needs to be adapted to only allow traffic from the same namespace or istio-system

3. If this works, you can also test path based routing

```
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ${INFERENCESERVICE}-external
spec:
  gateways:
    - kubeflow/kubeflow-gateway # migh have to be updated
  hosts:
    - '*'
  http:
    - match:
        - uri:
            prefix: /kserve/${NAMESPACE}/${INFERENCESERVICE}/
      rewrite:
        uri: /
      route:
        - destination:
            host: knative-local-gateway.istio-system.svc.cluster.local # might have to be updated
          headers:
            request:
              set:
                Host: ${INFERENCESERVICE}-predictor-default.${NAMESPACE}.svc.cluster.local
          weight: 100
      timeout: 300s
```

Which can then be called via
```
SERVICE_HOSTNAME=$INFERENCESERVICE.$NAMESPACE.example.com
GATEWAY="$KUBEFLOW_URL/kserve/${NAMESPACE}/$INFERENCESERVICE"
curl -H " bearer token ... \
    -H "Content-Type: application/json" \
    $GATEWAY/v1/models/kserve-iris:predict -d @iris-input.json
```

In the end the inferenceservice must only be available with a proper token from the default-editor service account as shown here
https://github.com/kubeflow/manifests/blob/c8bc6358ba715737267930219adf6a202f27e9c0/.github/workflows/pipeline_test.yaml#L86C1-L98C118

https://github.com/kubeflow/kubeflow/pull/6627 is also interesting
author_association: MEMBER
user: login: madmecodes
created_at: 2025-03-23T05:59:22Z
body: @juliusvonkohout 

i shall now make PR compatible with istio-cni also right? its currently setup with regular istio (init containers)

What i can understand right now:

**Before PR: Regular Istio (non-CNI)**

1. KServe endpoints were secured using the cluster-local-gateway with an ALLOW policy
2. This didn't properly integrate with Kubeflow's authentication system

**After PR: Still using regular Istio**

1. Changed the cluster-local-gateway to use oauth2-proxy authentication
2. Added specific AuthorizationPolicies for KServe endpoints
3. This properly secures KServe with the same authentication mechanism as other Kubeflow services

**Verification of Success:**

 **Success with valid token:**
* test_sklearn.py::test_sklearn_kserve PASSED
<img width="400" alt="Screenshot 2025-03-23 at 11 39 03 AM" src="https://github.com/user-attachments/assets/405f8f74-6190-4c56-b812-0edc48b66b29" />

**Failure without token (expected behavior):**
 * WARNING root:utils.py:105 M2M Token Not found, client authentication disabled.
* requests.exceptions.HTTPError: 403 Client Error: Forbidden
<img width="400" alt="Screenshot 2025-03-23 at 11 40 05 AM" src="https://github.com/user-attachments/assets/b2aecc5e-5740-4d92-bacd-e1864f22f634" />

**Proper redirection to login:**
* The HTML content in the response contains the Dex login page, confirming the oauth2-proxy integration is working.

**Also thing to consider**
restricting access by source namespace. As currently AuthorizationPolicy allows any authenticated request to access the paths, shall we restrict it to specific source namespaces like 


```
spec:
  rules:
    - from:
        - source:
            namespaces: ["kubeflow-user-example-com", "istio-system", "knative-serving"]
      to:
        - operation:
            paths:
              - /v1/models/*
              - /v2/models/*
```


**but now,**
Failing Dex workflow: Using Istio CNI
Which is part of a migration effort toward using CNI as the default (PR #3061)

<img width="500" alt="Screenshot 2025-03-23 at 11 24 05 AM" src="https://github.com/user-attachments/assets/b5757cbe-0678-43ef-bb09-669958ce77f4" />

author_association: CONTRIBUTOR
user: login: juliusvonkohout
created_at: 2025-03-23T12:39:36Z
body: Hello,

- [ ] the dex test is just a bit flaky. Lets try to make it more robust in this PR as well. I improved it a lot already, but we might need more or better timeouts in some places, such that it works reliably.
- [x] Alright let me enable istio-cni by default.
- [ ] do not hardcode "kubeflow-user-example-com" in the authorizationpolicy, but make it pick up the namespace where the authorizationpolicy is deployed in automatically.
- [ ] you need to make sure that you cannot inference with a token from another namespace e.g. kubeflow-user-example-com-attacker
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-03-23T13:18:00Z
body: @kromanow94 can you review as well?
author_association: MEMBER
user: login: madmecodes
created_at: 2025-03-24T12:06:08Z
body: working on Auth Policy, so that unwanted namespace get access denied 

PS: found
`In Istio, when a request enters through the ingress gateway (typically in the istio-system namespace), the source namespace evaluated by the AuthorizationPolicy is the namespace of the ingress gateway’s service account, not the namespace associated with the token in the Authorization header. Here’s why unauthorized access isn’t blocked`
author_association: CONTRIBUTOR
user: login: madmecodes
created_at: 2025-03-24T14:04:33Z
body: @juliusvonkohout  Currently Auth Policy is allowing access to models, via any namespace, if it has a valid JWT we need to change it to so that only specified namespaces in AuthPolicy can access them, am i thinking in the right direction?

i am testing it locally, when its too strict, it forbids all if too permissive, allows all (with valid JWT), working on a middle path, am i working on right problem?
<img width="500" alt="Screenshot 2025-03-24 at 7 32 24 PM" src="https://github.com/user-attachments/assets/44a3f01f-92b2-4a22-81ab-02e7b7a299f3" />

<img width="500" alt="Screenshot 2025-03-24 at 7 36 45 PM" src="https://github.com/user-attachments/assets/902fb042-70f5-4bba-a094-0875161365da" />
<img width="500" alt="Screenshot 2025-03-24 at 7 36 58 PM" src="https://github.com/user-attachments/assets/7d661953-7fed-4dc9-b1de-3f62cc714f21" />


author_association: CONTRIBUTOR
user: login: juliusvonkohout
created_at: 2025-03-24T15:04:18Z
body: > @juliusvonkohout Currently Auth Policy is allowing access to models, via any namespace, if it has a valid JWT we need to change it to so that only specified namespaces in AuthPolicy can access them, am i thinking in the right direction?
> 
> i am testing it locally, when its too strict, it forbids all if too permissive, allows all (with valid JWT), working on a middle path, am i working on right problem? <img alt="Screenshot 2025-03-24 at 7 32 24 PM" width="500" src="https://private-user-images.githubusercontent.com/137040550/426112167-44a3f01f-92b2-4a22-81ab-02e7b7a299f3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDI4Mjg4MjMsIm5iZiI6MTc0MjgyODUyMywicGF0aCI6Ii8xMzcwNDA1NTAvNDI2MTEyMTY3LTQ0YTNmMDFmLTkyYjItNGEyMi04MWFiLTAyZTdiN2EyOTlmMy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMzI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDMyNFQxNTAyMDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wZThkNDQzYTkxY2Y4MTMxODg5OTQ3NTJhMmZhMjkwMjZjNTFjNmRkNjg5Y2MyZTliYjFkMjlmOTJlZmRiZjFhJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.Qc1-IC_PKqwSKPkaph903Qla92wWs_ajt0sW50EPih8">
> <img alt="Screenshot 2025-03-24 at 7 36 45 PM" width="500" src="https://private-user-images.githubusercontent.com/137040550/426114133-902fb042-70f5-4bba-a094-0875161365da.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDI4Mjg4MjMsIm5iZiI6MTc0MjgyODUyMywicGF0aCI6Ii8xMzcwNDA1NTAvNDI2MTE0MTMzLTkwMmZiMDQyLTcwZjUtNGJiYS1hMDk0LTA4NzUxNjEzNjVkYS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMzI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDMyNFQxNTAyMDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jMjNhNzMzNDdlODIxY2JjZDEyOTUyNDYzMzBiMDA5OTFkODJhNWJlYmYxZThiZmJiZjgwMWY0ZTMxMjcxZjc3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.DZcJ_ys3H37qltJLN_R65GhZJTc4xp6eOLBS5hWnkJQ"> <img alt="Screenshot 2025-03-24 at 7 36 58 PM" width="500" src="https://private-user-images.githubusercontent.com/137040550/426114154-7d661953-7fed-4dc9-b1de-3f62cc714f21.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDI4Mjg4MjMsIm5iZiI6MTc0MjgyODUyMywicGF0aCI6Ii8xMzcwNDA1NTAvNDI2MTE0MTU0LTdkNjYxOTUzLTdmZWQtNGRjOS1iMWRlLTNmNjJjYzcxNGYyMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMzI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDMyNFQxNTAyMDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zYzAzNGZhMTU1MjczNjQxOGM1ZjliMDRiMWNiMTFhZjIwMWU1OWNhNzAyMTZhYWVlZmQxZWE0OTlkMTllOTcyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.ZmyaGanJvYpDqRNDAs5L-gnsYeMEN5jzJK8Z13zNfqA">

Yes, that is what we want. Only mebers/serviceaccounts of the same namespace should be able to access it. Maybe you can work with the allowing the default-editor, default and default-viewer serviceaccount of the same namespace as principals in the authorizationpolicy.
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-03-24T15:12:15Z
body: See also https://github.com/kubeflow/manifests/blob/master/apps/pipeline/upstream/base/installs/multi-user/istio-authorization-config.yaml
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-03-25T17:56:04Z
body: Let me do some changes, merge the PR and then you can create a follow-up PR for the remaining stuff @madmecodes 

- [ ] The dex test is just a bit flaky. Lets try to make it more robust.
- [ ] Do not hardcode "kubeflow-user-example-com" in the authorizationpolicy, but make it pick up the namespace where the authorizationpolicy is deployed in automatically.
- [ ] You need to make sure that you cannot inference with a token from another namespace e.g. kubeflow-user-example-com-attacker. For path-based and host-based access.
author_association: MEMBER
user: login: madmecodes
created_at: 2025-03-25T18:38:05Z
body: > Let me do some changes, merge the PR and then you can create a follow-up PR for the remaining stuff @madmecodes
> 
> * [ ]  The dex test is just a bit flaky. Lets try to make it more robust.
> * [ ]  Do not hardcode "kubeflow-user-example-com" in the authorizationpolicy, but make it pick up the namespace where the authorizationpolicy is deployed in automatically.
> * [ ]  You need to make sure that you cannot inference with a token from another namespace e.g. kubeflow-user-example-com-attacker. For path-based and host-based access.

yes, i am working on this only, facing some problems, troubleshooting them, i will create a follow up PR for these 3 things as mentioned once the PR is merged.
1. Make the Dex test more robust (fix flakiness)
2. Avoid hardcoding namespaces in AuthorizationPolicy
3. Ensure tokens from other namespaces can't access models
author_association: CONTRIBUTOR
user: login: juliusvonkohout
created_at: 2025-03-25T19:01:42Z
body: /lgtm
/approve
author_association: MEMBER
user: login: google-oss-prow[bot]
created_at: 2025-03-25T19:01:56Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubeflow/manifests/pull/3056#issuecomment-2752253762" title="Approved">juliusvonkohout</a>*

The full list of commands accepted by this bot can be found [here](https://oss.gprow.dev/command-help?repo=kubeflow%2Fmanifests).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubeflow/manifests/blob/master/OWNERS)~~ [juliusvonkohout]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE
user: login: juliusvonkohout
created_at: 2025-03-25T19:04:42Z
body: Please make sure that you rebase to master before you create the new branch / PR.
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-03-25T20:28:06Z
body:  In the follow up PR you can either remove or secure the cluster-local-gateway. Whatever works for kserve and keeps all  security tests green.
author_association: MEMBER