Repository: kubernetes-sigs/gateway-api-inference-extension

Issue:
Make Version specific sets of the getting started guide
<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**: it was mentioned: https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/577#issuecomment-2762763766 that we should version lock vllm in our getting started guide. We _do_ version lock in our releases, but that does _not_ make it to the site. We should remedy. 

**Why is this needed**:



Discussion:
@liu-cong: Is this a duplicate of https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/422? If not, can you clarify or consolidate these?

Solution patch:
diff --git a/config/charts/body-based-routing/templates/bbr.yaml b/config/charts/body-based-routing/templates/bbr.yaml
index 4b888dcb..e740e06e 100644
--- a/config/charts/body-based-routing/templates/bbr.yaml
+++ b/config/charts/body-based-routing/templates/bbr.yaml
@@ -19,7 +19,7 @@ spec:
         imagePullPolicy: {{ .Values.bbr.image.pullPolicy | default "Always" }}
         args:
         - "-streaming"
-        - "v"
+        - "-v"
         - "3"
         ports:
         - containerPort: 9004
diff --git a/config/charts/inferencepool/README.md b/config/charts/inferencepool/README.md
index 30087527..681fc783 100644
--- a/config/charts/inferencepool/README.md
+++ b/config/charts/inferencepool/README.md
@@ -9,20 +9,14 @@ To install an InferencePool named `vllm-llama3-8b-instruct`  that selects from e
 
 ```txt
 $ helm install vllm-llama3-8b-instruct ./config/charts/inferencepool \
-  --set inferencePool.name=vllm-llama3-8b-instruct \
   --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
-  --set inferencePool.targetPortNumber=8000
 ```
 
-where `inferencePool.targetPortNumber` is the pod that vllm backends served on and `inferencePool.modelServers.matchLabels` is the selector to match the vllm backends.
-
 To install via the latest published chart in staging  (--version v0 indicates latest dev version), you can run the following command:
 
 ```txt
 $ helm install vllm-llama3-8b-instruct \
-  --set inferencePool.name=vllm-llama3-8b-instruct \
   --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
-  --set inferencePool.targetPortNumber=8000 \
   oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/inferencepool --version v0
 ```
 
@@ -38,17 +32,17 @@ $ helm uninstall pool-1
 
 The following table list the configurable parameters of the chart.
 
-| **Parameter Name**                          | **Description**                                                                                                   |
-|---------------------------------------------|-------------------------------------------------------------------------------------------------------------------|
-| `inferencePool.name`                        | Name for the InferencePool, and inference extension will be named as `${inferencePool.name}-epp`.                |
-| `inferencePool.targetPortNumber`            | Target port number for the vllm backends, will be used to scrape metrics by the inference extension.             |
-| `inferencePool.modelServers.matchLabels`    | Label selector to match vllm backends managed by the inference pool.                                             |
-| `inferenceExtension.replicas`               | Number of replicas for the inference extension service. Defaults to `1`.                                           |
-| `inferenceExtension.image.name`             | Name of the container image used for the inference extension.                                                    |
-| `inferenceExtension.image.hub`              | Registry URL where the inference extension image is hosted.                                                     |
-| `inferenceExtension.image.tag`              | Image tag of the inference extension.                                                                             |
-| `inferenceExtension.image.pullPolicy`       | Image pull policy for the container. Possible values: `Always`, `IfNotPresent`, or `Never`. Defaults to `Always`. |
-| `inferenceExtension.extProcPort`            | Port where the inference extension service is served for external processing. Defaults to `9002`.                  |
+| **Parameter Name**                          | **Description**                                                                                                        |
+|---------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
+| `inferencePool.name`                        | Name for the InferencePool, and endpoint picker deployment and service will be named as `{.Release.name}-epp`.         |
+| `inferencePool.targetPortNumber`            | Target port number for the vllm backends, will be used to scrape metrics by the inference extension. Defaults to 8000. |
+| `inferencePool.modelServers.matchLabels`    | Label selector to match vllm backends managed by the inference pool.                                                   |
+| `inferenceExtension.replicas`               | Number of replicas for the endpoint picker extension service. Defaults to `1`.                                         |
+| `inferenceExtension.image.name`             | Name of the container image used for the endpoint picker.                                                              |
+| `inferenceExtension.image.hub`              | Registry URL where the endpoint picker image is hosted.                                                                |
+| `inferenceExtension.image.tag`              | Image tag of the endpoint picker.                                                                                      |
+| `inferenceExtension.image.pullPolicy`       | Image pull policy for the container. Possible values: `Always`, `IfNotPresent`, or `Never`. Defaults to `Always`.      |
+| `inferenceExtension.extProcPort`            | Port where the endpoint picker service is served for external processing. Defaults to `9002`.                          |
 
 ## Notes
 
diff --git a/config/charts/inferencepool/templates/NOTES.txt b/config/charts/inferencepool/templates/NOTES.txt
index 3d822165..22e5c0e1 100644
--- a/config/charts/inferencepool/templates/NOTES.txt
+++ b/config/charts/inferencepool/templates/NOTES.txt
@@ -1 +1 @@
-InferencePool {{ .Values.inferencePool.name }} deployed.
+InferencePool {{ .Release.Name }} deployed.
diff --git a/config/charts/inferencepool/templates/_helpers.tpl b/config/charts/inferencepool/templates/_helpers.tpl
index bb15f9e4..e011bb7c 100644
--- a/config/charts/inferencepool/templates/_helpers.tpl
+++ b/config/charts/inferencepool/templates/_helpers.tpl
@@ -12,7 +12,7 @@ app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
 Inference extension name
 */}}
 {{- define "gateway-api-inference-extension.name" -}}
-{{- $base := .Values.inferencePool.name | default "default-pool" | lower | trim | trunc 40 -}}
+{{- $base := .Release.Name | default "default-pool" | lower | trim | trunc 40 -}}
 {{ $base }}-epp
 {{- end -}}
 
@@ -20,5 +20,5 @@ Inference extension name
 Selector labels
 */}}
 {{- define "gateway-api-inference-extension.selectorLabels" -}}
-app: {{ include "gateway-api-inference-extension.name" . }}
+inferencepool: {{ include "gateway-api-inference-extension.name" . }}
 {{- end -}}
diff --git a/config/charts/inferencepool/templates/_validations.tpl b/config/charts/inferencepool/templates/_validations.tpl
index 55ed80c8..65c743b6 100644
--- a/config/charts/inferencepool/templates/_validations.tpl
+++ b/config/charts/inferencepool/templates/_validations.tpl
@@ -2,11 +2,6 @@
 common validations
 */}}
 {{- define "gateway-api-inference-extension.validations.inferencepool.common" -}}
-{{- if not $.Values.inferencePool.name }}
-{{- fail "missing .Values.inferencePool.name" }}
-{{- end }}
-
-
 {{- if or (empty $.Values.inferencePool.modelServers) (not $.Values.inferencePool.modelServers.matchLabels) }}
 {{- fail ".Values.inferencePool.modelServers.matchLabels is required" }}
 {{- end }}
diff --git a/config/charts/inferencepool/templates/epp-deployment.yaml b/config/charts/inferencepool/templates/epp-deployment.yaml
index ded9cb12..9faace73 100644
--- a/config/charts/inferencepool/templates/epp-deployment.yaml
+++ b/config/charts/inferencepool/templates/epp-deployment.yaml
@@ -22,7 +22,7 @@ spec:
         imagePullPolicy: {{ .Values.inferenceExtension.image.pullPolicy | default "Always" }}
         args:
         - -poolName
-        - {{ .Values.inferencePool.name }}
+        - {{ .Release.Name }}
         - -poolNamespace
         - {{ .Release.Namespace }}
         - -v
diff --git a/config/charts/inferencepool/templates/gke.yaml b/config/charts/inferencepool/templates/gke.yaml
new file mode 100644
index 00000000..86e8c4ff
--- /dev/null
+++ b/config/charts/inferencepool/templates/gke.yaml
@@ -0,0 +1,59 @@
+{{- if eq .Values.provider.name "gke" }}
+---
+kind: HealthCheckPolicy
+apiVersion: networking.gke.io/v1
+metadata:
+  name: {{ .Release.Name }}
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "gateway-api-inference-extension.labels" . | nindent 4 }}
+spec:
+  targetRef:
+    group: "inference.networking.x-k8s.io"
+    kind: InferencePool
+    name: {{ .Release.Name }}
+  default:
+    config:
+      type: HTTP
+      httpHealthCheck:
+          requestPath: /health
+          port:  {{ .Values.inferencePool.targetPortNumber }}
+---
+apiVersion: networking.gke.io/v1
+kind: GCPBackendPolicy
+metadata:
+  name: {{ .Release.Name }}
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "gateway-api-inference-extension.labels" . | nindent 4 }}
+spec:
+  targetRef:
+    group: "inference.networking.x-k8s.io"
+    kind: InferencePool
+    name: {{ .Release.Name }}
+  default:
+    timeoutSec: 300    # 5-minute timeout (adjust as needed)
+---
+apiVersion: monitoring.googleapis.com/v1
+kind: ClusterPodMonitoring
+metadata:
+  name: {{ .Release.Namespace }}-{{ .Release.Name }}
+  labels:
+    {{- include "gateway-api-inference-extension.labels" . | nindent 4 }}
+spec:
+  endpoints:
+  - port: metrics
+    scheme: http
+    interval: 5s
+    path: /metrics
+    authorization:
+      type: Bearer
+      credentials:
+        secret:
+          name: {{ .Values.gke.monitoringSecret }}
+          key: token
+          namespace: {{ .Release.Namespace }}
+  selector:
+    matchLabels:
+      {{- include "gateway-api-inference-extension.labels" . | nindent 8 }}
+{{- end }}
diff --git a/config/charts/inferencepool/templates/inferencepool.yaml b/config/charts/inferencepool/templates/inferencepool.yaml
index 2b79f399..4b279cbd 100644
--- a/config/charts/inferencepool/templates/inferencepool.yaml
+++ b/config/charts/inferencepool/templates/inferencepool.yaml
@@ -2,7 +2,7 @@
 apiVersion: inference.networking.x-k8s.io/v1alpha2
 kind: InferencePool
 metadata:
-  name: {{ .Values.inferencePool.name }}
+  name: {{ .Release.Name }}
   namespace: {{ .Release.Namespace }}
   labels:
     {{- include "gateway-api-inference-extension.labels" . | nindent 4 }}
diff --git a/config/charts/inferencepool/values.yaml b/config/charts/inferencepool/values.yaml
index 7b0c8f96..45dd11a1 100644
--- a/config/charts/inferencepool/values.yaml
+++ b/config/charts/inferencepool/values.yaml
@@ -8,8 +8,13 @@ inferenceExtension:
   extProcPort: 9002
 
 inferencePool:
-  # name: pool-1 # REQUIRED
   targetPortNumber: 8000
   # modelServers: # REQUIRED
     # matchLabels: 
     #   app: vllm-llama3-8b-instruct
+
+provider:
+  name: none
+
+gke:
+  monitoringSecret: inference-gateway-sa-metrics-reader-secret
diff --git a/config/manifests/gateway/enable_patch_policy.yaml b/config/manifests/gateway/enable_patch_policy.yaml
deleted file mode 100644
index 1e9818a1..00000000
--- a/config/manifests/gateway/enable_patch_policy.yaml
+++ /dev/null
@@ -1,27 +0,0 @@
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: envoy-gateway-config
-  namespace: envoy-gateway-system
-data:
-# This manifest's main purpose is to set `enabledEnvoyPatchPolicy` to `true`.
-# This only needs to be ran once on your cluster (unless you'd like to change anything. i.e. enabling the admin dash)
-# Any field under `admin` is optional, and only for enabling the admin endpoints, for debugging.
-# Admin Interface: https://www.envoyproxy.io/docs/envoy/latest/operations/admin
-# PatchPolicy docs: https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/#enable-envoypatchpolicy 
-  envoy-gateway.yaml: |
-    apiVersion: gateway.envoyproxy.io/v1alpha1
-    kind: EnvoyGateway
-    provider:
-      type: Kubernetes
-    gateway:
-      controllerName: gateway.envoyproxy.io/gatewayclass-controller
-    extensionApis:
-      enableEnvoyPatchPolicy: true      
-      enableBackend: true
-#    admin:
-#      enablePprof: true
-#      address:
-#        host: 127.0.0.1
-#        port: 19000
-#      enabledDumpConfig: true
diff --git a/config/manifests/gateway/gateway.yaml b/config/manifests/gateway/gateway.yaml
deleted file mode 100644
index 32f5d484..00000000
--- a/config/manifests/gateway/gateway.yaml
+++ /dev/null
@@ -1,50 +0,0 @@
-
----
-apiVersion: gateway.networking.k8s.io/v1
-kind: Gateway
-metadata:
-  name: inference-gateway
-spec:
-  gatewayClassName: inference-gateway
-  listeners:
-    - name: http
-      protocol: HTTP
-      port: 8080
-    - name: llm-gw
-      protocol: HTTP
-      port: 8081
----
-apiVersion: gateway.networking.k8s.io/v1
-kind: GatewayClass
-metadata:
-  name: inference-gateway
-spec:
-  controllerName: gateway.envoyproxy.io/gatewayclass-controller
----
-apiVersion: gateway.envoyproxy.io/v1alpha1
-kind: Backend
-metadata:
-  name: backend-dummy
-spec:
-  endpoints:
-    - fqdn:
-        # Both these values are arbitrary and unused as the PatchPolicy redirects requests.
-        hostname: 'foo.bar.com'
-        port: 8080
----
-apiVersion: gateway.networking.k8s.io/v1
-kind: HTTPRoute
-metadata:
-  name: llm-route
-spec:
-  parentRefs:
-    - name: inference-gateway
-      sectionName: llm-gw
-  rules:
-  - backendRefs:
-      - group: gateway.envoyproxy.io
-        kind: Backend
-        name: backend-dummy
-    timeouts:
-      request: "24h"
-      backendRequest: "24h"
diff --git a/config/manifests/gateway/gke/gateway.yaml b/config/manifests/gateway/gke/gateway.yaml
new file mode 100644
index 00000000..942cde5c
--- /dev/null
+++ b/config/manifests/gateway/gke/gateway.yaml
@@ -0,0 +1,10 @@
+kind: Gateway
+apiVersion: gateway.networking.k8s.io/v1
+metadata:
+ name: inference-gateway
+spec:
+ gatewayClassName: gke-l7-regional-external-managed
+ listeners:
+ - name: http
+   port: 80
+   protocol: HTTP
diff --git a/config/manifests/gateway/gke/gcp-backend-policy.yaml b/config/manifests/gateway/gke/gcp-backend-policy.yaml
new file mode 100644
index 00000000..519a5a93
--- /dev/null
+++ b/config/manifests/gateway/gke/gcp-backend-policy.yaml
@@ -0,0 +1,11 @@
+apiVersion: networking.gke.io/v1
+kind: GCPBackendPolicy
+metadata:
+  name: inferencepool-backend-policy
+spec:
+  targetRef:
+    group: "inference.networking.x-k8s.io"
+    kind: InferencePool
+    name: vllm-llama3-8b-instruct
+  default:
+    timeoutSec: 300
diff --git a/config/manifests/gateway/gke/healthcheck.yaml b/config/manifests/gateway/gke/healthcheck.yaml
new file mode 100644
index 00000000..95f4f2d2
--- /dev/null
+++ b/config/manifests/gateway/gke/healthcheck.yaml
@@ -0,0 +1,16 @@
+kind: HealthCheckPolicy
+apiVersion: networking.gke.io/v1
+metadata:
+  name: health-check-policy
+  namespace: default
+spec:
+  targetRef:
+    group: "inference.networking.x-k8s.io"
+    kind: InferencePool
+    name: vllm-llama2-7b
+  default:
+    config:
+      type: HTTP
+      httpHealthCheck:
+          requestPath: /health
+          port: 8000
diff --git a/config/manifests/gateway/httproute-with-timeout.yaml b/config/manifests/gateway/httproute-with-timeout.yaml
new file mode 100644
index 00000000..060f18c5
--- /dev/null
+++ b/config/manifests/gateway/httproute-with-timeout.yaml
@@ -0,0 +1,20 @@
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: llm-route
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: inference-gateway
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: vllm-llama2-7b
+    matches:
+    - path:
+        type: PathPrefix
+        value: /
+    timeouts:
+      request: 300s
diff --git a/config/manifests/gateway/httproute.yaml b/config/manifests/gateway/httproute.yaml
new file mode 100644
index 00000000..5bd8bfb6
--- /dev/null
+++ b/config/manifests/gateway/httproute.yaml
@@ -0,0 +1,18 @@
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: llm-route
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: inference-gateway
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: vllm-llama2-7b
+    matches:
+    - path:
+        type: PathPrefix
+        value: /
diff --git a/config/manifests/gateway/istio/destination-rule.yaml b/config/manifests/gateway/istio/destination-rule.yaml
new file mode 100644
index 00000000..f9cd0c3c
--- /dev/null
+++ b/config/manifests/gateway/istio/destination-rule.yaml
@@ -0,0 +1,10 @@
+apiVersion: networking.istio.io/v1
+kind: DestinationRule
+metadata:
+  name: epp-insecure-tls
+spec:
+  host: vllm-llama2-7b-epp
+  trafficPolicy:
+      tls:
+        mode: SIMPLE
+        insecureSkipVerify: true
diff --git a/config/manifests/gateway/istio/gateway.yaml b/config/manifests/gateway/istio/gateway.yaml
new file mode 100644
index 00000000..dd762678
--- /dev/null
+++ b/config/manifests/gateway/istio/gateway.yaml
@@ -0,0 +1,10 @@
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: inference-gateway
+spec:
+  gatewayClassName: istio
+  listeners:
+  - name: http
+    port: 80
+    protocol: HTTP
diff --git a/config/manifests/gateway/kgateway/gateway.yaml b/config/manifests/gateway/kgateway/gateway.yaml
new file mode 100644
index 00000000..7bcd08a6
--- /dev/null
+++ b/config/manifests/gateway/kgateway/gateway.yaml
@@ -0,0 +1,10 @@
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: inference-gateway
+spec:
+  gatewayClassName: kgateway
+  listeners:
+  - name: http
+    port: 80
+    protocol: HTTP
diff --git a/config/manifests/gateway/patch_policy.yaml b/config/manifests/gateway/patch_policy.yaml
deleted file mode 100644
index 923ce22c..00000000
--- a/config/manifests/gateway/patch_policy.yaml
+++ /dev/null
@@ -1,123 +0,0 @@
-apiVersion: gateway.envoyproxy.io/v1alpha1
-kind: EnvoyPatchPolicy
-metadata:
-  name: custom-response-patch-policy
-  namespace: default
-spec:
-  targetRef:
-    group: gateway.networking.k8s.io
-    kind: Gateway
-    name: inference-gateway
-  type: JSONPatch
-  jsonPatches:
-    # Necessary to create a cluster of the type: ORIGINAL_DST to allow for 
-    # direct pod scheduling. Which is heavily utilized in our scheduling.
-    # Specifically the field `original_dst_lb_config` allows us to enable
-    # `use_http_header` and `http_header_name`. 
-    # Source: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto
-    - type: "type.googleapis.com/envoy.config.cluster.v3.Cluster"
-      name: original_destination_cluster
-      operation:
-        op: add
-        path: ""
-        value:
-          name: original_destination_cluster
-          type: ORIGINAL_DST
-          original_dst_lb_config:
-            use_http_header: true
-            http_header_name: "x-gateway-destination-endpoint"
-          connect_timeout: 1000s
-          lb_policy: CLUSTER_PROVIDED
-          dns_lookup_family: V4_ONLY
-          circuit_breakers:
-            thresholds:
-            - max_connections: 40000
-              max_pending_requests: 40000
-              max_requests: 40000
-
-    # This ensures that envoy accepts untrusted certificates. We tried to explicitly
-    # set TrustChainVerification to ACCEPT_UNSTRUSTED, but that actually didn't work
-    # and what worked is setting the common_tls_context to empty.
-    - type: "type.googleapis.com/envoy.config.cluster.v3.Cluster"
-      name: "envoyextensionpolicy/default/ext-proc-policy/extproc/0"
-      operation:
-        op: add
-        path: "/transport_socket"
-        value:
-          name: "envoy.transport_sockets.tls"
-          typed_config:
-            "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext"
-            common_tls_context: {}
-    - type: "type.googleapis.com/envoy.config.route.v3.RouteConfiguration"
-      name: default/inference-gateway/llm-gw
-      operation:
-        op: replace
-        path: "/virtual_hosts/0/routes/0/route/cluster"
-        value: original_destination_cluster
-# Comment the below to disable full duplex streaming
-# NOTE: As of https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/552
-# FULL_DUPLEX_STREAMED is the primary supported protocol for ext-proc. The buffered variant is no longer
-# being actively developed, may be missing features/fixes, and will soon be removed.
-    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
-      name: "default/inference-gateway/llm-gw"
-      operation:
-        op: add
-        path: "/default_filter_chain/filters/0/typed_config/http_filters/0/typed_config/processing_mode/request_body_mode"
-        value: FULL_DUPLEX_STREAMED
-    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
-      name: "default/inference-gateway/llm-gw"
-      operation:
-        op: add
-        path: "/default_filter_chain/filters/0/typed_config/http_filters/0/typed_config/processing_mode/request_trailer_mode"
-        value: SEND
-    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
-      name: "default/inference-gateway/llm-gw"
-      operation:
-        op: add
-        path: "/default_filter_chain/filters/0/typed_config/http_filters/0/typed_config/processing_mode/response_body_mode"
-        value: FULL_DUPLEX_STREAMED
-    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
-      name: "default/inference-gateway/llm-gw"
-      operation:
-        op: replace
-        path: "/default_filter_chain/filters/0/typed_config/http_filters/0/typed_config/processing_mode/response_trailer_mode"
-        value: SEND
-    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
-      name: "default/inference-gateway/llm-gw"
-      operation:
-        op: replace
-        path: "/default_filter_chain/filters/0/typed_config/http_filters/0/typed_config/processing_mode/response_header_mode"
-        value: SEND
----
-apiVersion: gateway.envoyproxy.io/v1alpha1
-kind: EnvoyExtensionPolicy
-metadata:
-  name: ext-proc-policy
-  namespace: default
-spec:
-  extProc:
-    - backendRefs:
-      - group: ""
-        kind: Service
-        name: vllm-llama3-8b-instruct-epp
-        port: 9002
-      processingMode:
-        allowModeOverride: true
-        request:
-          body: Buffered
-        response:
-      # The timeouts are likely not needed here. We can experiment with removing/tuning them slowly.
-      # The connection limits are more important and will cause the opaque: ext_proc_gRPC_error_14 error in Envoy GW if not configured correctly. 
-      messageTimeout: 1000s
-      backendSettings:
-        circuitBreaker:
-          maxConnections: 40000
-          maxPendingRequests: 40000
-          maxParallelRequests: 40000
-        timeout:
-          tcp:
-            connectTimeout: 24h
-  targetRef:
-    group: gateway.networking.k8s.io
-    kind: HTTPRoute
-    name: llm-route
diff --git a/config/manifests/gateway/traffic_policy.yaml b/config/manifests/gateway/traffic_policy.yaml
deleted file mode 100644
index e110f173..00000000
--- a/config/manifests/gateway/traffic_policy.yaml
+++ /dev/null
@@ -1,16 +0,0 @@
-apiVersion: gateway.envoyproxy.io/v1alpha1
-kind: BackendTrafficPolicy
-metadata:
-  name: high-connection-route-policy
-spec:
-  targetRefs:
-  - group: gateway.networking.k8s.io
-    kind: HTTPRoute
-    name: llm-route
-  circuitBreaker:
-    maxConnections: 40000
-    maxPendingRequests: 40000
-    maxParallelRequests: 40000 
-  timeout:
-    tcp:
-      connectTimeout: 24h
\ No newline at end of file
diff --git a/config/manifests/inferencepool.yaml b/config/manifests/inferencepool-resources.yaml
similarity index 99%
rename from config/manifests/inferencepool.yaml
rename to config/manifests/inferencepool-resources.yaml
index 639157c1..d0f36e83 100644
--- a/config/manifests/inferencepool.yaml
+++ b/config/manifests/inferencepool-resources.yaml
@@ -22,6 +22,7 @@ spec:
     - protocol: TCP
       port: 9002
       targetPort: 9002
+      appProtocol: http2
   type: ClusterIP
 ---
 apiVersion: apps/v1
diff --git a/site-src/guides/index.md b/site-src/guides/index.md
index 99b78129..4548d5cd 100644
--- a/site-src/guides/index.md
+++ b/site-src/guides/index.md
@@ -1,9 +1,12 @@
 # Getting started with Gateway API Inference Extension
 
-This quickstart guide is intended for engineers familiar with k8s and model servers (vLLM in this instance). The goal of this guide is to get a first, single InferencePool up and running! 
+??? example "Experimental"
+
+    This project is still in an alpha state and breaking changes may occur in the future.
+
+This quickstart guide is intended for engineers familiar with k8s and model servers (vLLM in this instance). The goal of this guide is to get an Inference Gateway up and running! 
 
 ## **Prerequisites**
- - Envoy Gateway [v1.3.0](https://gateway.envoyproxy.io/docs/install/install-yaml/#install-with-yaml) or higher
  - A cluster with:
     - Support for services of type `LoadBalancer`. (This can be validated by ensuring your Envoy Gateway is up and running).
    For example, with Kind, you can follow [these steps](https://kind.sigs.k8s.io/docs/user/loadbalancer).
@@ -39,11 +42,10 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 
       This setup is using the formal `vllm-cpu` image, which according to the documentation can run vLLM on x86 CPU platform.
       For this setup, we use approximately 9.5GB of memory and 12 CPUs for each replica.  
-      While it is possible to deploy the model server with less resources, this is not recommended.  
-      For example, in our tests, loading the model using 8GB of memory and 1 CPU was possible but took almost 3.5 minutes and inference requests took unreasonable time.  
-      In general, there is a tradeoff between the memory and CPU we allocate to our pods and the performance. The more memory and CPU we allocate the better performance we can get.  
-      After running multiple configurations of these values we decided in this sample to use 9.5GB of memory and 12 CPUs for each replica, which gives reasonable response times. You can increase those numbers and potentially may even get better response times.
-      For modifying the allocated resources, adjust the numbers in `./config/manifests/vllm/cpu-deployment.yaml` as needed.  
+      
+      While it is possible to deploy the model server with less resources, this is not recommended. For example, in our tests, loading the model using 8GB of memory and 1 CPU was possible but took almost 3.5 minutes and inference requests took unreasonable time. In general, there is a tradeoff between the memory and CPU we allocate to our pods and the performance. The more memory and CPU we allocate the better performance we can get.
+      
+      After running multiple configurations of these values we decided in this sample to use 9.5GB of memory and 12 CPUs for each replica, which gives reasonable response times. You can increase those numbers and potentially may even get better response times. For modifying the allocated resources, adjust the numbers in [cpu-deployment.yaml](https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/cpu-deployment.yaml) as needed.  
 
       Deploy a sample vLLM deployment with the proper protocol to work with the LLM Instance Gateway.
       ```bash
@@ -52,68 +54,180 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 
 ### Install the Inference Extension CRDs
 
-   ```bash
-   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/crd/bases/inference.networking.x-k8s.io_inferencepools.yaml
-   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/crd/bases/inference.networking.x-k8s.io_inferencemodels.yaml
-   ```
-   
+=== "Latest Release"
+
+      ```bash
+      VERSION=v0.2.0
+      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/$VERSION/manifests.yaml
+      ```
+
+=== "Dev Version"
+
+      ```bash
+      kubectl apply -k https://github.com/kubernetes-sigs/gateway-api-inference-extension/config/crd
+      ```
+
 ### Deploy InferenceModel
 
    Deploy the sample InferenceModel which is configured to load balance traffic between the `food-review-0` and `food-review-1`
    [LoRA adapters](https://docs.vllm.ai/en/latest/features/lora.html) of the sample model server.
+
    ```bash
    kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencemodel.yaml
    ```
 
-### Update Envoy Gateway Config to enable Patch Policy**
+### Deploy the InferencePool and Extension
 
-   Our custom LLM Gateway ext-proc is patched into the existing envoy gateway via `EnvoyPatchPolicy`. To enable this feature, we must extend the Envoy Gateway config map. To do this, simply run:
    ```bash
-   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/enable_patch_policy.yaml
-   kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system
+   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencepool-resources.yaml
    ```
-   Additionally, if you would like to enable the admin interface, you can uncomment the admin lines and run this again.
 
-### Deploy Gateway
+### Deploy Inference Gateway
 
-   ```bash
-   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gateway.yaml
-   ```
-   > **_NOTE:_** This file couples together the gateway infra and the HTTPRoute infra for a convenient, quick startup. Creating additional/different InferencePools on the same gateway will require an additional set of: `Backend`, `HTTPRoute`, the resources included in the `./config/manifests/gateway/ext-proc.yaml` file, and an additional `./config/manifests/gateway/patch_policy.yaml` file. ***Should you choose to experiment, familiarity with xDS and Envoy are very useful.***
+   Choose one of the following options to deploy an Inference Gateway.
 
-   Confirm that the Gateway was assigned an IP address and reports a `Programmed=True` status:
-   ```bash
-   $ kubectl get gateway inference-gateway
-   NAME                CLASS               ADDRESS         PROGRAMMED   AGE
-   inference-gateway   inference-gateway   <MY_ADDRESS>    True         22s
-   ```
-### Deploy the InferencePool and Extension
+=== "GKE"
 
-   ```bash
-   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencepool.yaml
-   ```
-### Deploy Envoy Gateway Custom Policies
+      1. Enable the Gateway API and configure proxy-only subnets when necessary. See [Deploy Gateways](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways) 
+      for detailed instructions.
 
-   ```bash
-   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/patch_policy.yaml
-   ```
-   > **_NOTE:_** This is also per InferencePool, and will need to be configured to support the new pool should you wish to experiment further.
-   
-### **OPTIONALLY**: Apply Traffic Policy
+      1. Deploy Gateway and HealthCheckPolicy resources
+
+         ```bash
+         kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/gateway.yaml
+         kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/healthcheck.yaml
+         ```
+
+         Confirm that the Gateway was assigned an IP address and reports a `Programmed=True` status:
+         ```bash
+         $ kubectl get gateway inference-gateway
+         NAME                CLASS               ADDRESS         PROGRAMMED   AGE
+         inference-gateway   inference-gateway   <MY_ADDRESS>    True         22s
+         ```
+
+=== "Istio"
+
+      Please note that this feature is currently in an experimental phase and is not intended for production use. 
+      The implementation and user experience are subject to changes as we continue to iterate on this project.
+
+      1.  Requirements
+
+         - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
+
+      1. Install Istio
+      
+         ```
+         TAG=1.26-alpha.80c74f7f43482c226f4f4b10b4dda6261b67a71f
+         # on Linux
+         wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-linux-amd64.tar.gz
+         tar -xvf istioctl-$TAG-linux-amd64.tar.gz
+         # on macOS
+         wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-osx.tar.gz
+         tar -xvf istioctl-$TAG-osx.tar.gz
+         # on Windows
+         wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-win.zip
+         unzip istioctl-$TAG-win.zip
+
+         ./istioctl install --set tag=$TAG --set hub=gcr.io/istio-testing
+         ```
+
+      1. If you run the Endpoint Picker (EPP) with the `--secureServing` flag set to `true` (the default mode), it is currently using a self-signed certificate. As a security measure, Istio does not trust self-signed certificates by default. As a temporary workaround, you can apply the destination rule to bypass TLS verification for EPP. A more secure TLS implementation in EPP is being discussed in [Issue 582](https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/582).
+
+         ```bash
+         kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/destination-rule.yaml
+         ```
+
+      1. Deploy Gateway
+
+         ```bash
+         kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/gateway.yaml
+         ```
+
+      1. Label the gateway
+
+         ```bash
+         kubectl label gateway llm-gateway istio.io/enable-inference-extproc=true
+         ```
 
-   For high-traffic benchmarking you can apply this manifest to avoid any defaults that can cause timeouts/errors.
+         Confirm that the Gateway was assigned an IP address and reports a `Programmed=True` status:
+         ```bash
+         $ kubectl get gateway inference-gateway
+         NAME                CLASS               ADDRESS         PROGRAMMED   AGE
+         inference-gateway   inference-gateway   <MY_ADDRESS>    True         22s
+         ```
+
+=== "Kgateway"
+
+      [Kgateway](https://kgateway.dev/) v2.0.0 adds support for inference extension as a **technical preview**. This means do not
+      run Kgateway with inference extension in production environments. Refer to [Issue 10411](https://github.com/kgateway-dev/kgateway/issues/10411)
+      for the list of caveats, supported features, etc.
+
+      1. Requirements
+
+         - [Helm](https://helm.sh/docs/intro/install/) installed.
+         - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
+
+      1. Install Kgateway CRDs
+
+         ```bash
+         helm upgrade -i --create-namespace --namespace kgateway-system --version $VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
+         ```
+
+      1. Install Kgateway
+
+         ```bash
+         helm upgrade -i --namespace kgateway-system --version $VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway
+         --set inferenceExtension.enabled=true
+         ```
+
+      1. Deploy Gateway
+
+         ```bash
+         kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kgateway/gateway.yaml
+         ```
+
+         Confirm that the Gateway was assigned an IP address and reports a `Programmed=True` status:
+         ```bash
+         $ kubectl get gateway inference-gateway
+         NAME                CLASS               ADDRESS         PROGRAMMED   AGE
+         inference-gateway   kgateway            <MY_ADDRESS>    True         22s
+         ```
+
+### Deploy the HTTPRoute
 
    ```bash
-   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/traffic_policy.yaml
+   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/httproute.yaml
    ```
 
+### Configure Timeouts
+
+   Given that default timeouts for above implementations may be insufficient for most inference workloads, it is recommended to configure a timeout appropriate for your intended use case.
+
+=== "GKE"
+
+      ```bash
+      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/gcp-backend-policy.yaml
+      ```
+
+=== "Istio"
+
+      ```bash
+      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/httproute-with-timeout.yaml
+      ```
+
+=== "Kgateway"
+
+      ```bash
+      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/httproute-with-timeout.yaml
+      ```
+
 ### Try it out
 
    Wait until the gateway is ready.
 
    ```bash
    IP=$(kubectl get gateway/inference-gateway -o jsonpath='{.status.addresses[0].value}')
-   PORT=8081
+   PORT=80
 
    curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
    "model": "food-review",
@@ -126,18 +240,32 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 ### Cleanup
 
    The following cleanup assumes you would like to clean ALL resources that were created in this quickstart guide.  
-   please be careful not to delete resources you'd like to keep.
-   ```bash
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/traffic_policy.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/extension_policy.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/patch_policy.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencepool.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gateway.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/enable_patch_policy.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencemodel.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/crd/bases/inference.networking.x-k8s.io_inferencepools.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/crd/bases/inference.networking.x-k8s.io_inferencemodels.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/cpu-deployment.yaml --ignore-not-found
-   kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/gpu-deployment.yaml --ignore-not-found
-   kubectl delete secret hf-token --ignore-not-found
-   ```
+   Please be careful not to delete resources you'd like to keep.
+
+   1. Uninstall the Inference Pool
+
+      ```bash
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencepool-resources.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencemodel.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/cpu-deployment.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/gpu-deployment.yaml --ignore-not-found
+      kubectl delete secret hf-token --ignore-not-found
+      ```
+
+   1. Uninstall the Gateway
+
+      ```bash
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/gateway.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/healthcheck.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/gcp-backend-policy.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/gateway.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/destination-rule.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kgateway/gateway.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/httproute.yaml --ignore-not-found
+      ```
+
+   1. Uninstall the CRDs
+
+      ```bash
+      kubectl delete -k https://github.com/kubernetes-sigs/gateway-api-inference-extension/config/crd --ignore-not-found
+      ```
diff --git a/tools/dynamic-lora-sidecar/Dockerfile b/tools/dynamic-lora-sidecar/Dockerfile
index 4f6c743e..4faf360c 100644
--- a/tools/dynamic-lora-sidecar/Dockerfile
+++ b/tools/dynamic-lora-sidecar/Dockerfile
@@ -2,7 +2,7 @@ FROM python:3.9-slim-buster AS test
 
 WORKDIR /dynamic-lora-reconciler-test
 COPY requirements.txt .
-COPY sidecar/* . 
+COPY sidecar/* ./ 
 RUN pip install -r requirements.txt
 RUN python -m unittest discover || exit 1  
 
@@ -18,6 +18,6 @@ RUN pip install --upgrade pip
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
 
-COPY sidecar/* . 
+COPY sidecar/* ./
 
 CMD ["python", "sidecar.py"]
\ No newline at end of file
diff --git a/tools/dynamic-lora-sidecar/README.md b/tools/dynamic-lora-sidecar/README.md
index be05f9e9..f14dbfc7 100644
--- a/tools/dynamic-lora-sidecar/README.md
+++ b/tools/dynamic-lora-sidecar/README.md
@@ -29,21 +29,34 @@ The sidecar uses the vLLM server's API to load or unload adapters based on the c
 
 ## Usage
 
+
 1. **Build the Docker Image:**
    ```bash
    docker build -t <your-image-name> .
+   ```
+
 2. **Create a configmap:**
-    ```bash
-    kubectl create configmap name-of-your-configmap --from-file=your-file.yaml
+   ```bash
+   kubectl create configmap name-of-your-configmap --from-file=your-file.yaml
+   ```
+
 3. **Mount the configmap and configure sidecar in your pod**
-    ```yaml
-    volumeMounts: # DO NOT USE subPath
-          - name: config-volume
-            mountPath:  /config
-    ```
-    Do not use subPath, since configmap updates are not reflected in the file
+   ```yaml
+   volumeMounts: # DO NOT USE subPath
+         - name: config-volume
+           mountPath:  /config
+   ```
+   Do not use subPath, since configmap updates are not reflected in the file
 
-[deployment]: deployment.yaml it uses [sidecar](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/)(`initContainer` with `restartPolicy` set to `always`) which is beta feature enabled by default since k8s version 1.29. They need to be enabled in 1.28 and prior to 1.28 sidecar are not officially supported.
+## Command Line Arguments
+
+The sidecar supports the following command-line arguments:
+
+- `--health-check-timeout`: Maximum time in seconds to wait for the vLLM server health check (default: 300)
+- `--health-check-interval`: Interval in seconds between health check attempts (default: 2)
+- `--reconcile-trigger`: Time in seconds between forced reconciliation runs (default: 5)
+- `--config`: Path to the config map file (default: value from DYNAMIC_LORA_ROLLOUT_CONFIG env var or "/config/configmap.yaml")
+- `--config-validation`: Enable config validation (default: True)
 
 ## Configuration Fields
 - `vLLMLoRAConfig`[**required**]  base key 
@@ -61,11 +74,41 @@ The sidecar uses the vLLM server's API to load or unload adapters based on the c
         -  `source`[**required**] path (remote or local) to lora adapter
         - `base-model`[*optional*] Base model for lora adapter
 
-
-
+## Example Deployment
+
+The [deployment.yaml](deployment.yaml) file shows an example of deploying the sidecar with custom parameters:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dynamic-lora-reconciler
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: dynamic-lora-reconciler
+  template:
+    metadata:
+      labels:
+        app: dynamic-lora-reconciler
+    spec:
+      containers:
+      - name: reconciler
+        image: your-image:tag
+        command: ["python", "sidecar.py", "--health-check-timeout", "600", "--health-check-interval", "5", "--reconcile-trigger", "10"] #optional if overriding default values
+        volumeMounts:
+        - name: config-volume
+          mountPath: /config
+      volumes:
+      - name: config-volume
+        configMap:
+          name: name-of-your-configmap
+```
+
+Note: This uses [sidecar](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/)(`initContainer` with `restartPolicy` set to `always`) which is beta feature enabled by default since k8s version 1.29. They need to be enabled in 1.28 and prior to 1.28 sidecar are not officially supported.
 
 ## Screenshots & Testing
 The sidecar was tested with the Deployment and ConfigMap specified in this repo. Here are screen grabs of the logs from the sidecar and vllm server. One can verify that the adapters were loaded by querying `v1/models` and looking at vllm logs.
-![lora-adapter-syncer](screenshots/lora-syncer-sidecar.png)
-![config map change](screenshots/configmap-change.png)
+![lora-adapter-syncer](screenshots/lora-syncer-logs.png)
 ![vllm-logs](screenshots/vllm-logs.png)
diff --git a/tools/dynamic-lora-sidecar/screenshots/configmap-change.png b/tools/dynamic-lora-sidecar/screenshots/configmap-change.png
deleted file mode 100644
index e17f060b..00000000
Binary files a/tools/dynamic-lora-sidecar/screenshots/configmap-change.png and /dev/null differ
diff --git a/tools/dynamic-lora-sidecar/screenshots/lora-syncer-logs.png b/tools/dynamic-lora-sidecar/screenshots/lora-syncer-logs.png
new file mode 100644
index 00000000..2cd7a4d6
Binary files /dev/null and b/tools/dynamic-lora-sidecar/screenshots/lora-syncer-logs.png differ
diff --git a/tools/dynamic-lora-sidecar/screenshots/lora-syncer-sidecar.png b/tools/dynamic-lora-sidecar/screenshots/lora-syncer-sidecar.png
deleted file mode 100644
index c7b90253..00000000
Binary files a/tools/dynamic-lora-sidecar/screenshots/lora-syncer-sidecar.png and /dev/null differ
diff --git a/tools/dynamic-lora-sidecar/sidecar/sidecar.py b/tools/dynamic-lora-sidecar/sidecar/sidecar.py
index 02070f3f..00de99e3 100644
--- a/tools/dynamic-lora-sidecar/sidecar/sidecar.py
+++ b/tools/dynamic-lora-sidecar/sidecar/sidecar.py
@@ -1,6 +1,7 @@
 import requests
 import yaml
 import time
+import argparse
 from jsonschema import validate
 from watchfiles import awatch
 from dataclasses import dataclass
@@ -30,18 +31,35 @@ def current_time_human() -> str:
     return now.strftime("%Y-%m-%d %H:%M:%S %Z%z")
 
 
+def parse_arguments():
+    """Parse command line arguments."""
+    parser = argparse.ArgumentParser(description='vLLM LoRA Adapter Reconciler')
+    parser.add_argument('--health-check-timeout', type=int, default=300,
+                        help='Health check timeout in seconds (default: 300)')
+    parser.add_argument('--health-check-interval', type=int, default=2,
+                        help='Health check interval in seconds (default: 2)')
+    parser.add_argument('--reconcile-trigger', type=int, default=5,
+                        help='Reconciliation trigger interval in seconds (default: 5)')
+    parser.add_argument('--config', type=str, default=CONFIG_MAP_FILE,
+                        help=f'Path to config map file (default: {CONFIG_MAP_FILE})')
+    parser.add_argument('--config-validation', action='store_true', default=True,
+                        help='Enable config validation (default: True)')
+    return parser.parse_args()
+
+
 class FileChangeHandler(FileSystemEventHandler):
     """Custom event handler that handles file modifications."""
 
-    def __init__(self, reconciler):
+    def __init__(self, reconciler, config_file):
         super().__init__()
         self.reconciler = reconciler
+        self.config_file = config_file
 
     def on_modified(self, event):
         logging.info("modified!")
-        logging.info(f"Config '{CONFIG_MAP_FILE}' modified!")
+        logging.info(f"Config '{self.config_file}' modified!")
         self.reconciler.reconcile()
-        logging.info(f"model server reconcile to Config '{CONFIG_MAP_FILE}' !")
+        logging.info(f"model server reconcile to Config '{self.config_file}' !")
 
 
 @dataclass
@@ -65,10 +83,17 @@ class LoraReconciler:
     Reconciles adapters registered on vllm server with adapters listed in configmap in current state
     """
 
-    def __init__(self, config_validation=True):
-        self.health_check_timeout = datetime.timedelta(seconds=300)
-        self.health_check_interval = datetime.timedelta(seconds=15)
+    def __init__(self, config_file, health_check_timeout, health_check_interval, 
+                 reconcile_trigger_seconds, config_validation=True):
+        self.config_file = config_file
         self.config_validation = config_validation
+        self.health_check_timeout = datetime.timedelta(seconds=health_check_timeout)
+        self.health_check_interval = datetime.timedelta(seconds=health_check_interval)
+        self.reconcile_trigger_seconds = reconcile_trigger_seconds
+        
+        logging.info(f"Settings initialized: health check timeout={health_check_timeout}s, "
+                     f"interval={health_check_interval}s, "
+                     f"reconcile trigger={self.reconcile_trigger_seconds}s")
 
     def validate_config(self, c) -> bool:
         try:
@@ -77,14 +102,14 @@ class LoraReconciler:
                 validate(instance=c, schema=schema)
                 return True
         except Exception as e:
-            logging.error(f"Cannot load config {CONFIG_MAP_FILE} validation error: {e}")
+            logging.error(f"Cannot load config {self.config_file} validation error: {e}")
             return False
 
     @property
     def config(self):
         """Load configmap into memory"""
         try:
-            with open(CONFIG_MAP_FILE, "r") as f:
+            with open(self.config_file, "r") as f:
                 c = yaml.safe_load(f)
                 if self.config_validation and not self.validate_config(c):
                     return {}
@@ -93,7 +118,7 @@ class LoraReconciler:
                 c = c.get("vLLMLoRAConfig", {})
                 return c
         except Exception as e:
-            logging.error(f"cannot load config {CONFIG_MAP_FILE} {e}")
+            logging.error(f"cannot load config {self.config_file} {e}")
             return {}
 
     @property
@@ -215,8 +240,9 @@ class LoraReconciler:
     def reconcile(self):
         """Reconciles model server with current version of configmap"""
         logging.info(
-            f"reconciling model server {self.model_server} with config stored at {CONFIG_MAP_FILE}"
+            f"reconciling model server {self.model_server} with config stored at {self.config_file}"
         )
+        
         if not self.is_server_healthy:
             logging.error(f"vllm server at {self.model_server} not healthy")
             return
@@ -240,21 +266,40 @@ class LoraReconciler:
 
 
 async def main():
-    reconciler_instance = LoraReconciler()
-    logging.info(f"Running initial reconcile for config map {CONFIG_MAP_FILE}")
+    args = parse_arguments()
+    
+    # Update CONFIG_MAP_FILE with argument value
+    config_file = args.config
+    
+    reconciler_instance = LoraReconciler(
+        config_file=config_file,
+        health_check_timeout=args.health_check_timeout,
+        health_check_interval=args.health_check_interval,
+        reconcile_trigger_seconds=args.reconcile_trigger,
+        config_validation=args.config_validation
+    )
+    
+    logging.info(f"Running initial reconcile for config map {config_file}")
     reconciler_instance.reconcile()
 
-    event_handler = FileChangeHandler(reconciler_instance)
+    event_handler = FileChangeHandler(reconciler_instance, config_file)
     observer = Observer()
     observer.schedule(
-        event_handler, path=os.path.dirname(CONFIG_MAP_FILE), recursive=False
+        event_handler, path=os.path.dirname(config_file), recursive=False
     )
     observer.start()
 
     try:
-        logging.info(f"Starting to watch {CONFIG_MAP_FILE} for changes...")
+        logging.info(f"Starting to watch {config_file} for changes and performing periodic reconciliation...")
         while True:
-            await asyncio.sleep(1)
+            # Get current trigger interval from reconciler
+            trigger_seconds = reconciler_instance.reconcile_trigger_seconds
+            logging.info(f"Waiting {trigger_seconds}s before next reconciliation...")
+            # Wait for configured trigger interval
+            await asyncio.sleep(trigger_seconds)
+            # Force trigger reconciliation
+            logging.info("Periodic reconciliation triggered")
+            reconciler_instance.reconcile()
     except KeyboardInterrupt:
         logging.info("Stopped by user.")
         observer.stop()
@@ -262,4 +307,4 @@ async def main():
 
 
 if __name__ == "__main__":
-    asyncio.run(main())
+    asyncio.run(main())
\ No newline at end of file
diff --git a/tools/dynamic-lora-sidecar/sidecar/test_sidecar.py b/tools/dynamic-lora-sidecar/sidecar/test_sidecar.py
index 6f7e447f..59a60e6b 100644
--- a/tools/dynamic-lora-sidecar/sidecar/test_sidecar.py
+++ b/tools/dynamic-lora-sidecar/sidecar/test_sidecar.py
@@ -2,8 +2,10 @@ import unittest
 from unittest.mock import patch, Mock, mock_open, call
 import yaml
 import os
-from sidecar import LoraReconciler, CONFIG_MAP_FILE, BASE_FIELD, LoraAdapter
+import datetime
+from sidecar import LoraReconciler, LoraAdapter, CONFIG_MAP_FILE, BASE_FIELD
 
+# Update TEST_CONFIG_DATA to include the new configuration parameters
 TEST_CONFIG_DATA = {
     BASE_FIELD: {
         "host": "localhost",
@@ -49,13 +51,14 @@ TEST_CONFIG_DATA = {
         },
     }
 }
+
 EXIST_ADAPTERS = [
-    LoraAdapter(a["id"], a["base-model"], a["source"])
+    LoraAdapter(a["id"], a["source"], a["base-model"])
     for a in TEST_CONFIG_DATA[BASE_FIELD]["ensureExist"]["models"]
 ]
 
 NOT_EXIST_ADAPTERS = [
-    LoraAdapter(a["id"], a["base-model"], a["source"])
+    LoraAdapter(a["id"], a["source"], a["base-model"])
     for a in TEST_CONFIG_DATA[BASE_FIELD]["ensureNotExist"]["models"]
 ]
 RESPONSES = {
@@ -101,7 +104,15 @@ class LoraReconcilerTest(unittest.TestCase):
             mock_response = getMockResponse()
             mock_response.json.return_value = RESPONSES["v1/models"]
             mock_get.return_value = mock_response
-            self.reconciler = LoraReconciler(False)
+            
+            # Create reconciler with command line argument values instead of config file values
+            self.reconciler = LoraReconciler(
+                config_file=CONFIG_MAP_FILE,
+                health_check_timeout=180,
+                health_check_interval=10,
+                reconcile_trigger_seconds=30,
+                config_validation=False
+            )
             self.maxDiff = None
 
     @patch("sidecar.requests.get")
@@ -167,20 +178,47 @@ class LoraReconcilerTest(unittest.TestCase):
                         mock_get_response.json.return_value = RESPONSES["v1/models"]
                         mock_get.return_value = mock_get_response
                         mock_post.return_value = getMockResponse()
-                        self.reconciler = LoraReconciler()
-                        self.reconciler.reconcile()
                         
-                        # 1 adapter is in both exist and not exist list, only 2 are expected to be loaded
-                        mock_load.assert_has_calls(
-                            calls=[call(EXIST_ADAPTERS[0]), call(EXIST_ADAPTERS[2])]
+                        # Create reconciler with command line argument values
+                        self.reconciler = LoraReconciler(
+                            config_file=CONFIG_MAP_FILE,
+                            health_check_timeout=180,
+                            health_check_interval=10,
+                            reconcile_trigger_seconds=30,
+                            config_validation=False
                         )
-                        assert mock_load.call_count == 2
+                        self.reconciler.reconcile()
                         
-                        # 1 adapter is in both exist and not exist list, only 2 are expected to be unloaded
-                        mock_unload.assert_has_calls(
-                            calls=[call(NOT_EXIST_ADAPTERS[0]), call(NOT_EXIST_ADAPTERS[2])]
-                        )
-                        assert mock_unload.call_count == 2
+                        # First check the call count
+                        self.assertEqual(mock_load.call_count, 2, "Expected 2 load adapter calls")
+                        self.assertEqual(mock_unload.call_count, 2, "Expected 2 unload adapter calls")
+                        
+                        # Check that the adapters with the correct IDs were loaded
+                        loaded_ids = [call.args[0].id for call in mock_load.call_args_list]
+                        self.assertIn("sql-lora-v1", loaded_ids, "sql-lora-v1 should have been loaded")
+                        self.assertIn("already_exists", loaded_ids, "already_exists should have been loaded")
+                        
+                        # Check that the adapters with the correct IDs were unloaded
+                        unloaded_ids = [call.args[0].id for call in mock_unload.call_args_list]
+                        self.assertIn("sql-lora-v2", unloaded_ids, "sql-lora-v2 should have been unloaded")
+                        self.assertIn("to_remove", unloaded_ids, "to_remove should have been unloaded")
+
+    def test_health_check_settings(self):
+        """Test that health check settings are properly initialized from command line args"""
+        # Create reconciler with specific values
+        reconciler = LoraReconciler(
+            config_file=CONFIG_MAP_FILE,
+            health_check_timeout=240,
+            health_check_interval=15,
+            reconcile_trigger_seconds=45,
+            config_validation=False
+        )
+        
+        # Check that values are properly set
+        self.assertEqual(reconciler.health_check_timeout, datetime.timedelta(seconds=240))
+        self.assertEqual(reconciler.health_check_interval, datetime.timedelta(seconds=15))
+        self.assertEqual(reconciler.reconcile_trigger_seconds, 45)
+
 
 if __name__ == "__main__":
-    unittest.main()
+    unittest.main()
\ No newline at end of file


Test patch:
diff --git a/test/e2e/epp/e2e_suite_test.go b/test/e2e/epp/e2e_suite_test.go
index f9dea1cc..643bbf75 100644
--- a/test/e2e/epp/e2e_suite_test.go
+++ b/test/e2e/epp/e2e_suite_test.go
@@ -75,7 +75,7 @@ const (
 	// inferModelManifest is the manifest for the inference model CRD.
 	inferModelManifest = "../../../config/crd/bases/inference.networking.x-k8s.io_inferencemodels.yaml"
 	// inferExtManifest is the manifest for the inference extension test resources.
-	inferExtManifest = "../../../config/manifests/inferencepool.yaml"
+	inferExtManifest = "../../../config/manifests/inferencepool-resources.yaml"
 	// envoyManifest is the manifest for the envoy proxy test resources.
 	envoyManifest = "../../testdata/envoy.yaml"
 	// modelServerManifestFilepathEnvVar is the env var that holds absolute path to the manifest for the model server test resource.