Repository: stolostron/acm-with-llm
Pull request #10: Improve prompt to get OperatorPolicy
Author: login: yiraeChristineKim
Labels: name: approved
name: lgtm
Created: 2025-03-25T13:20:03Z | Merged: 2025-03-26T01:05:39Z

**Description**
When users request an OperatorPolicy, OpenAI (including ChatGPT) often provides incorrect answers or defaults to a ConfigurationPolicy instead. This prompt ensures more accurate OperatorPolicy responses, enhancing the user experience with ACM policies.


Diff:
diff --git a/env-sample b/env-sample
index f8719b3..1c30552 100644
--- a/env-sample
+++ b/env-sample
@@ -7,3 +7,4 @@ DBHOST=localhost
 DBPORT=5432
 DBSCHEMA=public
 
+ACM_RELEASED_VERSION=2.12
diff --git a/src/acm_agents.py b/src/acm_agents.py
index f816f3d..5fb5e2b 100644
--- a/src/acm_agents.py
+++ b/src/acm_agents.py
@@ -53,6 +53,57 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+https://raw.githubusercontent.com/stolostron/config-policy-controller/refs/tags/v{acm_released_version}.0/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml
+Follow the example to ensure accuracy and alignment.
+```yaml
+apiVersion: policy.open-cluster-management.io/v1beta1
+kind: OperatorPolicy
+metadata:
+  name: example-operator
+  labels:
+    policy.open-cluster-management.io/cluster-name: "managed"
+    policy.open-cluster-management.io/cluster-namespace: "managed"
+spec:
+  remediationAction: enforce # optional
+  severity: medium # optional
+  complianceType: musthave # required
+  complianceConfig: # optional
+    catalogSourceUnhealthy: Compliant
+    DeploymentsUnavailable: NonCompliant
+    UpgradesAvailable: Compliant
+    DeprecationsPresent: Compliant
+  operatorGroup: # optional
+    name: grc-dep-group
+    namespace: operator-policy-testns
+    targetNamespaces:
+      - operator-policy-testns
+  subscription: # required
+    name: example-operator
+    namespace: grc-dep
+    channel: alpha
+    source: grc-mock-source
+    sourceNamespace: olm
+  upgradeApproval: Automatic # required
+  removalBehavior: # optional
+    operatorGroups: DeleteIfUnused
+    subscriptions: Delete
+    clusterServiceVersions: Delete
+    customResourceDefinitions: Delete
+  versions: # optional
+    - example-operator.v0.36.0
+
+```
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
+
+If any discrepancies arise between the CRD and examples, revise and adjust the response accordingly.
+Ensure that the OperatorPolicy includes all required fields, especially upgradeApproval, and subscription.
+Do not mix in outdated structures, and always ensure that the response aligns with the latest specification.
+
 You can write a ACM policy yaml given a task as below 
 If a user gives you some feedback on the yaml you have produced,
 process it, think through it and improve the yaml. 
@@ -97,7 +148,8 @@ class AgentState(TypedDict):
     print("⚠️ Warning: Failed to load .env file!")
     sys.exit(1)
 
-openai.api_key  = os.getenv('OPENAI_API_KEY')
+acm_released_version = os.getenv('ACM_RELEASED_VERSION')
+openai.api_key = os.getenv('OPENAI_API_KEY')
 #model = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
 #gpt-4o
 #gpt-4o-mini
@@ -165,7 +217,7 @@ def author_node(state: AgentState):
     print( "Iteration number: ",state['iteration']+1)
     
     messages = [
-        SystemMessage(content=AUTHOR_PROMPT.format(content=state['task'])),
+        SystemMessage(content=AUTHOR_PROMPT.format(content=state['task'], acm_released_version=acm_released_version)),
         HumanMessage(content=state['summary'])
     ]
     


Reviews:
user: login: saswatamcode
body: Thanks, looks good generally, couple of comments.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-25T14:45:13Z
user: login: yiraeChristineKim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T14:48:50Z
user: login: saswatamcode
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T14:51:51Z
user: login: yiraeChristineKim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T14:52:06Z
user: login: yiraeChristineKim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T14:54:21Z
user: login: saswatamcode
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T14:57:14Z
user: login: yiraeChristineKim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T15:06:52Z
user: login: saswatamcode
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T15:10:36Z
user: login: yiraeChristineKim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T15:18:55Z
user: login: yiraeChristineKim
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T19:05:01Z
user: login: saswatamcode
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T19:48:57Z
user: login: saswatamcode
body: /lgtm
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-25T19:49:55Z
user: login: bjoydeep
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T01:00:16Z
user: login: bjoydeep
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-26T01:00:59Z

Review comments:
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
path: src/acm_agents.py
user: login: saswatamcode
body: Do we need the examples of field names? I feel like this might push the model to always use them (even when not strictly needed)

How about only mentioning the required fields (as done below)
created_at: 2025-03-25T14:43:32Z
author_association: MEMBER
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
+- Follow the example in https://github.com/stolostron/config-policy-controller/blob/main/test/resources/case38_operator_install/deprecation/all.yaml to ensure accuracy and alignment.
+
+If any discrepancies arise between the CRD and examples, revise and adjust the response accordingly.
+Ensure that the OperatorPolicy includes all required fields, especially upgradeApproval, and subscription.
+Do not mix in outdated structures, and always ensure that the response aligns with the latest specification.
path: src/acm_agents.py
user: login: saswatamcode
body: I wonder if pinning version in prompt would help, instead of latest.
created_at: 2025-03-25T14:44:45Z
author_association: MEMBER
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
+- Follow the example in https://github.com/stolostron/config-policy-controller/blob/main/test/resources/case38_operator_install/deprecation/all.yaml to ensure accuracy and alignment.
+
+If any discrepancies arise between the CRD and examples, revise and adjust the response accordingly.
+Ensure that the OperatorPolicy includes all required fields, especially upgradeApproval, and subscription.
+Do not mix in outdated structures, and always ensure that the response aligns with the latest specification.
path: src/acm_agents.py
user: login: yiraeChristineKim
body: Which version you mean? 
created_at: 2025-03-25T14:48:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
+- Follow the example in https://github.com/stolostron/config-policy-controller/blob/main/test/resources/case38_operator_install/deprecation/all.yaml to ensure accuracy and alignment.
+
+If any discrepancies arise between the CRD and examples, revise and adjust the response accordingly.
+Ensure that the OperatorPolicy includes all required fields, especially upgradeApproval, and subscription.
+Do not mix in outdated structures, and always ensure that the response aligns with the latest specification.
path: src/acm_agents.py
user: login: saswatamcode
body: The version of the OperatorPolicy CRD? I think the repo has release tags
created_at: 2025-03-25T14:51:51Z
author_association: MEMBER
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
path: src/acm_agents.py
user: login: yiraeChristineKim
body: so for now, we don't have much example of Operatorpolicy in github. eventhough I added CRD, it doesn't understand well. AI forgot required fields all the times or give me wrong structure. For example it add subscription as object-template. also added operatorGroup under object-template I guess it learn with configuration policy.
created_at: 2025-03-25T14:52:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
+- Follow the example in https://github.com/stolostron/config-policy-controller/blob/main/test/resources/case38_operator_install/deprecation/all.yaml to ensure accuracy and alignment.
+
+If any discrepancies arise between the CRD and examples, revise and adjust the response accordingly.
+Ensure that the OperatorPolicy includes all required fields, especially upgradeApproval, and subscription.
+Do not mix in outdated structures, and always ensure that the response aligns with the latest specification.
path: src/acm_agents.py
user: login: yiraeChristineKim
body: I plan to replace https://github.com/stolostron/config-policy-controller/blob/main/test/resources/case38_operator_install/deprecation/all.yaml with a more complete version in the future. Currently, best practice OperatorPolicies have not been added to the policy collection, but I will be adding many OperatorPolicies soon.
created_at: 2025-03-25T14:54:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
path: src/acm_agents.py
user: login: saswatamcode
body: Yeah it's tricky, I think feeding it a complete example might be the best here, (or passing a complete YAML directly, although that might be costly token-wise). That way it won't forget stuff
created_at: 2025-03-25T14:57:14Z
author_association: MEMBER
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
path: src/acm_agents.py
user: login: yiraeChristineKim
body: If we add yaml here, then everytime CRD change, do we have to update the yaml?

created_at: 2025-03-25T15:06:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
path: src/acm_agents.py
user: login: saswatamcode
body: Honestly, kind of why I would love to have some YAML "RAG" repo, which would have examples of such CRDs, that we can maintain/update, and pass dynamically to LLM based on user prompt.
iirc, openshift lightspeed does a similar thing.

Not blocking this PR btw, this should work better than earlier :)
created_at: 2025-03-25T15:10:35Z
author_association: MEMBER
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
path: src/acm_agents.py
user: login: yiraeChristineKim
body: I can add yaml for temporary util our team add examples of OperatorPolicy in Policy-collection or we add RAG repo. Thanks for your advice!
created_at: 2025-03-25T15:18:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
path: src/acm_agents.py
user: login: yiraeChristineKim
body: @saswatamcode I pushed a new commit, Adding yaml gives more accurate results. I guess AI don't understand CRD well for now. Thanks @saswatamcode for your advice!! 
created_at: 2025-03-25T19:05:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
path: src/acm_agents.py
user: login: saswatamcode
body: That's great! 🙂 
created_at: 2025-03-25T19:48:57Z
author_association: MEMBER
diff_hunk: @@ -53,6 +53,19 @@ class AgentState(TypedDict):
 for Red Hat Advanced Cluster Management (RHACM or ACM). 
 You are aware of the different kind (CRDs) under apiVersion: policy.open-cluster-management.io 
 and know how to use them.
+
+When a user asks about `OperatorPolicy`, always refer strictly to the `OperatorPolicy` resource as defined in the Open Cluster Management CRD at
+ https://github.com/open-cluster-management-io/config-policy-controller/blob/main/deploy/crds/policy.open-cluster-management.io_operatorpolicies.yaml.
+
+- Do not use `ConfigurationPolicy` or any other policy types.
+- Ensure compliance with the structure outlined in the CRD.
+- Use the correct field names as defined in the CRD, such as `operatorGroup`, `subscription`, `remediationAction`, and `complianceType`.
path: src/acm_agents.py
user: login: bjoydeep
body: I think this will get a little easier when we move to a llama-stack. That change is coming soon.
created_at: 2025-03-26T01:00:16Z
author_association: COLLABORATOR

PR comments:
user: login: yiraeChristineKim
created_at: 2025-03-25T14:39:54Z
body: @bjoydeep 
author_association: CONTRIBUTOR
user: login: yiraeChristineKim
created_at: 2025-03-25T14:40:02Z
body: /cc @bjoydeep 
author_association: CONTRIBUTOR
user: login: yiraeChristineKim
created_at: 2025-03-25T14:40:44Z
body: /cc @saswatamcode 
author_association: CONTRIBUTOR
user: login: bjoydeep
created_at: 2025-03-26T01:00:37Z
body: /lgtm
author_association: COLLABORATOR
user: login: openshift-ci[bot]
created_at: 2025-03-26T01:01:04Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/stolostron/acm-with-llm/pull/10#pullrequestreview-2715571911" title="Approved">bjoydeep</a>*, *<a href="https://github.com/stolostron/acm-with-llm/pull/10#pullrequestreview-2715011076" title="LGTM">saswatamcode</a>*, *<a href="https://github.com/stolostron/acm-with-llm/pull/10#" title="Author self-approved">yiraeChristineKim</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=stolostron%2Facm-with-llm).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/stolostron/acm-with-llm/blob/main/OWNERS)~~ [bjoydeep]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE