MLflowによる大規模言語モデルの評価

以前こちらの記事を書きました。

https://qiita.com/taka_yayoi/items/f06adbf5510703b0510b

マニュアルにも掲載されています。気づいたらMLflowもバージョン2.9.1になってました。

https://docs.databricks.com/ja/mlflow/llm-evaluate.html

こちらのノートブックをウォークスルーします。

https://mlflow.org/docs/latest/llms/llm-evaluate/notebooks/question-answering-evaluation.html

```py
import os
os.environ["OPENAI_API_KEY"] = dbutils.secrets.get("demo-token-takaaki.yayoi", "openai")
```

```py
import openai
import pandas as pd

import mlflow
```

```py
eval_df = pd.DataFrame(
    {
        "inputs": [
            "How does useEffect() work?",
            "What does the static keyword in a function mean?",
            "What does the 'finally' block in Python do?",
            "What is the difference between multiprocessing and multithreading?",
        ],
        "ground_truth": [
            "The useEffect() hook tells React that your component needs to do something after render. React will remember the function you passed (we’ll refer to it as our “effect”), and call it later after performing the DOM updates.",
            "Static members belongs to the class, rather than a specific instance. This means that only one instance of a static member exists, even if you create multiple objects of the class, or if you don't create any. It will be shared by all objects.",
            "'Finally' defines a block of code to run when the try... except...else block is final. The finally block will be executed no matter if the try block raises an error or not.",
            "Multithreading refers to the ability of a processor to execute multiple threads concurrently, where each thread runs a process. Whereas multiprocessing refers to the ability of a system to run multiple processors in parallel, where each processor can run one or more threads.",
        ],
    }
)
```

```py
with mlflow.start_run() as run:
    system_prompt = "Answer the following question in two sentences"
    basic_qa_model = mlflow.openai.log_model(
        model="gpt-3.5-turbo",
        task=openai.ChatCompletion,
        artifact_path="model",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": "{question}"},
        ],
    )
    results = mlflow.evaluate(
        basic_qa_model.model_uri,
        eval_df,
        targets="ground_truth",  # specify which column corresponds to the expected output
        model_type="question-answering",  # model type indicates which metrics are relevant for this task
        evaluators="default",
    )
results.metrics
```

```
{'toxicity/v1/mean': 0.00021154171190573834,
 'toxicity/v1/variance': 2.705208305975482e-09,
 'toxicity/v1/p90': 0.0002699144184589386,
 'toxicity/v1/ratio': 0.0,
 'exact_match/v1': 0.0}
```

メトリクスに関してはこちらに説明がありました。

https://mlflow.org/docs/latest/llms/llm-evaluate/index.html#use-default-metrics-for-pre-defined-model-types

https://huggingface.co/spaces/evaluate-measurement/toxicity

```py
results.tables["eval_results_table"]
```
![Screenshot 2023-12-14 at 11.47.19.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1168882/6f5549b2-59c1-046d-41e3-189eef63f5b3.png)

評価ビューでも確認できました。
![Screenshot 2023-12-14 at 11.47.47.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1168882/b189d269-7b0c-b157-0919-ba7439cc3dac.png)

と思ったら、すでに包括的に評価されていらっしゃいました。脱帽です。

https://qiita.com/isanakamishiro2/items/8b6214b8dde113cee389

### Databricksクイックスタートガイド

[Databricksクイックスタートガイド](https://www.amazon.co.jp/dp/B09V1YXFVQ/)


### Databricks無料トライアル

[Databricks無料トライアル](https://databricks.com/jp/try-databricks)