watsonx.governanceでwatsonx.ai以外（3rd Party）のRAGの監視

# PROMPTE TEMPLATEをSPACEにPROMOTE＆DEPLOY

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/3a5ea6e6-e6dc-4093-9a81-2baaca1e4e41.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/4cd7fb6d-1aca-4090-a47b-af026c03ab84.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/c4334fa0-d615-489f-aa26-6415152d89ba.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/42c8e6fe-557e-42b6-91f7-12a4837ac759.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/70f89447-4220-4d95-b5be-93d8add1c5ac.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/5489608e-7f8b-428b-a199-0c5e9819b9f1.png)

# Evaluateを有効化

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/db54145e-f581-4d2d-b180-2c4a2139df49.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/28b5d9f5-858c-470a-be7c-45aed197e28f.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/5005149d-d4cb-4b45-8d0c-08f0f62105f0.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/a59b93e8-7b1f-4d80-8db0-efd75214601a.png)

# SPACE_IDとSUBSCRIPTION_IDの取得

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/170165a5-9c8e-49ad-aeab-420464fd6e2c.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/a00d135b-82e8-4580-a2cb-47bf613be53b.png)

# Payloadデータの送信

Payloadデータ、コンテキストと質問、それらをもとにLLMで生成された回答などと言った入出力データ。ここでは正解は含まれてない。Payloadデータの送信＆評価はステージが<bold>production（実働）</bold>である必要がある。例えば、生成AIアプリケーションの開発ステージをProductionにして、生成AIのアプリケーションの入出力データをwatsonx.governanceに送付しれば、定期的に評価され、それらを基に生成AIアプリのモニタリングが可能になる。

```python
# %%
PROJECT_ID = "<INPUT YOUR PROJECT_ID>"
PROMPT_TEMPLATE_ASSET_ID = "<INPUT YOUR PROMPT_TEMPLATE_ASSET_ID>"

# %%
CPD_URL = "<INPUT YOUR CPD_URL>"
CPD_USERNAME = "<INPUT YOUR CPD_USERNAME>"
CPD_PASSWORD = "<INPUT YOUR CPD_PASSWORD>"
CPD_API_KEY = "<INPUT YOUR CPD_API_KEY>"

# %%
SPACE_ID = "<INPUT YOUR SPACE_ID>"
SUBSCRIPTION_ID = "<INPUT YOUR SUBSCRIPTION_ID>"

# %%
from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator
from ibm_watson_openscale import *
from ibm_watson_openscale.supporting_classes.enums import *
from ibm_watson_openscale.supporting_classes import *

authenticator = CloudPakForDataAuthenticator(
    url=CPD_URL,
    username=CPD_USERNAME,
    password=CPD_PASSWORD,
    disable_ssl_verification=True
)

wos_client = APIClient(
    service_url=CPD_URL,
    authenticator=authenticator,
)
data_mart_id = wos_client.service_instance_id
print(data_mart_id)
print(wos_client.version)

# %%
wos_client.monitor_instances.show(target_target_id=SUBSCRIPTION_ID)

# %%
monitor_definition_id = "mrm"
response = wos_client.monitor_instances.list(
   data_mart_id=data_mart_id,
   monitor_definition_id=monitor_definition_id,
   target_target_id=SUBSCRIPTION_ID,
   space_id=SPACE_ID)
response.result.to_dict()
# %%
monitor_instance_id = response.result.to_dict()["monitor_instances"][0]["metadata"]["id"]
print(monitor_instance_id)

# %%
import csv

context_fields = ["context1", "context2", "context3"]
question_field = "question"
feature_fields = context_fields + [question_field]
prediction = "generated_text"

pl_data = []
prediction_list = []

test_data_path = "test.csv"
with open(test_data_path, 'r') as csv_file:
    csv_reader = csv.DictReader(csv_file)
    for row in csv_reader:
        request = {
            "parameters": {
                "template_variables": {
                }
            }
        }
        for each in feature_fields:
            request["parameters"]["template_variables"][each] = str(row[each])

        predicted_val = row[prediction]
        prediction_list.append(predicted_val)
        response = {
            "results": [
                {
                    prediction: predicted_val
                }
            ]
        }
        record = {"request": request, "response": response}
        pl_data.append(record)
pl_data

# %%
import json

print(json.dumps(pl_data, indent=2, ensure_ascii=False))

# %%
from ibm_watson_openscale.supporting_classes.enums import *

data_set_id = None
response = wos_client.data_sets.list(
    type=DataSetTypes.PAYLOAD_LOGGING, 
    target_target_id=SUBSCRIPTION_ID, 
    target_target_type=TargetTypes.SUBSCRIPTION)
response.result.to_dict()

# %%
data_set_id = response.result.to_dict()['data_sets'][0]['metadata']['id']
print(data_set_id)

# %%
response = wos_client.data_sets.store_records(
   data_set_id=data_set_id,
   request_body=pl_data,
   background_mode=True)
response.result.to_dict()

# %%
wos_client.data_sets.get_records_count(data_set_id=data_set_id)

# %%

```

# Payloadデータによる手動評価

Payloadとして送付された入出力データを利用して1時間毎に評価するけど、GUIで手動評価も可能。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/7f3dc122-25a9-4729-bd7f-5aba737de40d.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/570963f3-6680-4edb-ad6b-9bb9d6d7281d.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/33aa6fa2-4dae-46c4-ac23-8534a0a7848e.png)


# Feedbackデータの送信

Feedbackデータ、コンテキストと質問、それらをもとにLLMで生成された回答などと言った入出力データに正解含まれたもの。Feedbackデータの送信＆評価はステージがproduction（実働）である必要がある。例えば、生成AIアプリをProductionにして、生成AIのアプリケーションの入出力データと正解をwatsonx.governanceに送付しれば、定期的に評価され、それらを基に生成AIアプリのモニタリングが可能になる。

```python
# %%
PROJECT_ID = "<INPUT YOUR PROJECT_ID>"
PROMPT_TEMPLATE_ASSET_ID = "<INPUT YOUR PROMPT_TEMPLATE_ASSET_ID>"

# %%
CPD_URL = "<INPUT YOUR CPD_URL>"
CPD_USERNAME = "<INPUT YOUR CPD_USERNAME>"
CPD_PASSWORD = "<INPUT YOUR CPD_PASSWORD>"
CPD_API_KEY = "<INPUT YOUR CPD_API_KEY>"

# %%
SPACE_ID = "<INPUT YOUR SPACE_ID>"
SUBSCRIPTION_ID = "<INPUT YOUR SUBSCRIPTION_ID>"

# %%
from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator
from ibm_watson_openscale import *
from ibm_watson_openscale.supporting_classes.enums import *
from ibm_watson_openscale.supporting_classes import *

authenticator = CloudPakForDataAuthenticator(
    url=CPD_URL,
    username=CPD_USERNAME,
    password=CPD_PASSWORD,
    disable_ssl_verification=True
)

wos_client = APIClient(
    service_url=CPD_URL,
    authenticator=authenticator,
)
data_mart_id = wos_client.service_instance_id
print(data_mart_id)
print(wos_client.version)

# %%
wos_client.monitor_instances.show(target_target_id=SUBSCRIPTION_ID)

# %%
monitor_definition_id = "mrm"
response = wos_client.monitor_instances.list(
   data_mart_id=data_mart_id,
   monitor_definition_id=monitor_definition_id,
   target_target_id=SUBSCRIPTION_ID,
   space_id=SPACE_ID)
response.result.to_dict()
# %%
monitor_instance_id = response.result.to_dict()["monitor_instances"][0]["metadata"]["id"]
print(monitor_instance_id)

# %%
from ibm_watson_openscale.supporting_classes.enums import *

data_set_id = None
response = wos_client.data_sets.list(
    type=DataSetTypes.FEEDBACK, 
    target_target_id=SUBSCRIPTION_ID, 
    target_target_type=TargetTypes.SUBSCRIPTION)
response.result.to_dict()

# %%
data_set_id = response.result.to_dict()['data_sets'][0]['metadata']['id']
print(data_set_id)

# %%
import pandas as pd

test_data_path = "test.csv"
llm_data = pd.read_csv(test_data_path)
llm_data.head()

# %%
llm_data.shape

# %%
test_data_content = []
context_fields = ["context1", "context2", "context3"]
question_field = "question"
feature_fields = context_fields + [question_field]   #For Alternative Dataset
prediction_list = llm_data["generated_text"].tolist()
label_column = "answer"

for _, row in llm_data.iterrows():
    # Read each row from the DataFrame and add label and prediction values
    result_row = [row[key] for key in feature_fields if key in row]
    result_row.append(row[label_column])
    result_row.append(row["generated_text"])
    test_data_content.append(result_row)

if len(test_data_content) == 10: # 10 records are there in the downloaded CSV
    print("generated feedback data from DataFrame")
else:
    print("Failed to generate feedback data from DataFrame, kindly verify the DataFrame content")

fields = feature_fields.copy()
fields.append(label_column)
fields.append("_original_prediction")
feedback_data = [
    {
        "fields": fields,
        "values": test_data_content
    }
]

feedback_data
# %%
import json

print(json.dumps(feedback_data, indent=2, ensure_ascii=False))

# %%
response = wos_client.data_sets.store_records(
    data_set_id=data_set_id,
    request_body=feedback_data,
    background_mode=True)
response.result.to_dict()

# %%
wos_client.data_sets.get_records_count(data_set_id=data_set_id)

# %%

```


# Feedbackデータによる手動評価

Payloadと同じで、Feedbackデータとして送付された入出力データを利用して1時間毎に評価するけど、GUIで手動評価も可能。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/655c4c59-ca75-4ef9-8152-4abd0d29dcd9.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3756172/f67d092c-85c7-4f54-a8f3-fb002bd03e56.png)




# 考察というか、watsonx.governanceの整理
開発フェーズでは、まず PROJECT に PROMPT TEMPLATE を作成し、watsonx.governance で評価を行う。
評価結果に一定の目処がついたら、SPACE に PRODUCTION としてデプロイする。
本番稼働中の Web アプリケーションでは、入出力データを watsonx.governance に送信することで、定期的な評価が可能となる。
ユーザーはこれをモニタリングすることで、生成AIのユースケースを継続的に監視・改善できる。

次は、watsonx.governanceでモニタリングしているRAGのデモを作ってみよう。。。