Python SDK から gpt-4o-miniをファインチューニング(Azure OpenAI サービス)

Python SDKでgpt-4o-miniのQAタスクのファインチューニングしてみます。
基本的には以下の方法に従っています。

https://learn.microsoft.com/ja-jp/azure/ai-services/openai/how-to/fine-tuning?pivots=programming-language-python

前に書いた記事ではAzure AI FoundryからGUIでファインチューニングしましたが、当記事ではPythonから実行します。

https://qiita.com/FukuharaYohei/items/53064460264cdf43b272

# 処理
## 0. 前提
[記事「Azure AI Foundry から gpt-4o-miniをファインチューニング」]()の「0. 前提」および「1. 訓練/検証ファイル準備」まで完了していることが前提です。

## 1. 環境変数設定

**あとで確認**

後に作成するPythonから読み込むために`.env`ファイルを作成して、以下の２つの環境変数を設定します。

```text:.env
AZURE_OPENAI_API_KEY=<API Key>
AZURE_OPENAI_ENDPOINT=<End Point>
```

## 2. Pythonプログラム作成
Jupyterで実行しています。

### 環境
|種類|バージョン|備考|
|:-:|:-:|:-:|
|Python(Pyenv)|3.10.2|古いのに特に意味はないです。たまたますぐ使えた環境
|Ubuntu(WSL2)|22.04.5 LTS|
|Poetry|1.7.1|

### Python 追加パッケージ

|種類|バージョン|備考|
|:-:|:-:|:-:|
|openai|1.63.2|
|requests|2.32.3|
|jupyterlab|4.3.5|

### Pythonプログラム
#### 1. パッケージインポート
```python: パッケージインポート
import json

from openai import AzureOpenAI
```

#### 2. JSONLファイル読込
ファイルは「[Azure AI Foundry から gpt-4o-miniをファインチューニング](https://qiita.com/FukuharaYohei/items/53064460264cdf43b272)」で使ったものと同じです。

```python: JSONLファイル読込
DIR = './sample_data/'
TRAIN_FILE = DIR + 'train.jsonl'
VALIDATION_FILE = DIR + 'validation.jsonl'

def read_jsonl(file_name: str) -> list:
    with open(file_name, 'r', encoding='utf-8') as f:
        dataset = [json.loads(line) for line in f]

    # Dataset stats
    print(f"Number of examples in {file_name}: {len(dataset)}")
    print(f"First example in {file_name}:")
    for message in dataset[0]["messages"]:
        print(message)
    return dataset

train_dataset = read_jsonl(TRAIN_FILE)
print("----------")
validation_dataset = read_jsonl(VALIDATION_FILE)
```

```bash
Number of examples in ./sample_data/train.jsonl: 80
First example in ./sample_data/train.jsonl:
{'role': 'system', 'content': '情報を見つけるのに役立つ AI アシスタントです。'}
{'role': 'user', 'content': '福原洋平さんの職業は何ですか？'}
{'role': 'assistant', 'content': '福原洋平さんは都市計画コンサルタント兼環境デザイナーとして活躍しています。  '}

----------
Number of examples in ./sample_data/validation.jsonl: 20
First example in ./sample_data/validation.jsonl:
{'role': 'system', 'content': '情報を見つけるのに役立つ AI アシスタントです。'}
{'role': 'user', 'content': '福原洋平さんがプロジェクトで最初に行う作業は？'}
{'role': 'assistant', 'content': '現地調査と住民の意見収集を最初に行います。  '}
```

#### 3. ファイルをアップロード
ファイルをアップロードします。面倒だったので、End Point と API Keyは環境変数から読み込まず固定値渡しています。
以下を参考にしてください。

https://learn.microsoft.com/ja-jp/azure/ai-services/openai/tutorials/fine-tune?tabs=python-new%2Ccommand-line#retrieve-key-and-endpoint

```python: ファイルアップロード
client = AzureOpenAI(
  azure_endpoint = "<End Point>",
  api_key = "<API Key>",
  api_version = "2024-08-01-preview"  # This API version or later is required to access seed/events/checkpoint features
)

# Upload the training and validation dataset files to Azure OpenAI with the SDK.
training_response = client.files.create(
    file = open(TRAIN_FILE, "rb"), purpose="fine-tune"
)

validation_response = client.files.create(
    file = open(VALIDATION_FILE, "rb"), purpose="fine-tune"
)

print("Training file ID:", training_response.id)
print("Validation file ID:", validation_response.id)
```

```bash
Training file ID: file-79600873356049e084973a13ea571a40
Validation file ID: file-637528dab95e418f97956a625a424960
```

Azure AI Foundryからもアップロードしたファイルを確認可能(実行後のresults.csvを画面上見えているが、この時点では未作成)。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/134273/b371f328-959e-48fe-ad18-81ca6e4e90d4.png)

#### 4. ファインチューニング実行
まずはジョブ実行をします。

```python: ファインチューニング実行
# Submit fine-tuning training job
response = client.fine_tuning.jobs.create(
    training_file = training_response.id,
    validation_file = validation_response.id,
    model = "gpt-4o-mini", # Enter base model name. Note that in Azure OpenAI the model name contains dashes and cannot contain dot/period characters.
    seed = 105 # seed parameter controls reproducibility of the fine-tuning job. If no seed is specified one will be generated automatically.
)

# responseはこの後他の変数を入れるので退避
job_id = response.id

# You can use the job ID to monitor the status of the fine-tuning job.
# The fine-tuning job will take some time to start and complete.
print("Job ID:", job_id)
print("Status:", response.status)
print(response.model_dump_json(indent=2))
```

```bash
Job ID: ftjob-d78af615cda84674b541c026e764bcb7
Status: pending
{
  "id": "ftjob-d78af615cda84674b541c026e764bcb7",
  "created_at": 1740444480,
  "error": null,
  "fine_tuned_model": null,
  "finished_at": null,
  "hyperparameters": {
    "batch_size": -1,
    "learning_rate_multiplier": 1.0,
    "n_epochs": -1
  },
  "model": "gpt-4o-mini-2024-07-18",
  "object": "fine_tuning.job",
  "organization_id": null,
  "result_files": null,
  "seed": 105,
  "status": "pending",
  "trained_tokens": null,
  "training_file": "file-79600873356049e084973a13ea571a40",
  "validation_file": "file-637528dab95e418f97956a625a424960",
  "estimated_finish": 1740445382,
  "integrations": null,
  "method": null
}
```

#### 5. ジョブ実行ステータス確認
ジョブ実行をJupyter上で確認します。

```python:
# Track training status
from IPython.display import clear_output
import time

start_time = time.time()

# Get the status of our fine-tuning job.
response = client.fine_tuning.jobs.retrieve(job_id)

status = response.status

# If the job isn't done yet, poll it every 10 seconds.
while status not in ["succeeded", "failed"]:
    time.sleep(10)

    response = client.fine_tuning.jobs.retrieve(job_id)
    print(response.model_dump_json(indent=2))
    print("Elapsed time: {} minutes {} seconds".format(int((time.time() - start_time) // 60), int((time.time() - start_time) % 60)))
    status = response.status
    print(f'Status: {status}')
    clear_output(wait=True)

print(f'Fine-tuning job {job_id} finished with status: {status}')

# List all fine-tuning jobs for this resource.
print('Checking other fine-tune jobs for this resource.')
response = client.fine_tuning.jobs.list()
print(f'Found {len(response.data)} fine-tune jobs.')
```

ジョブが終わると以下のような出力がされます。

```bash
Fine-tuning job ftjob-d78af615cda84674b541c026e764bcb7 finished with status: succeeded
Checking other fine-tune jobs for this resource.
Found 1 fine-tune jobs.
```

#### 6. 実行結果確認
実行結果を確認します。
グラフィカルに確認したい場合には、当然Azure AI Foundryからの方が見やすいです

```python:
# イベント確認
response = client.fine_tuning.jobs.list_events(fine_tuning_job_id=job_id, limit=10)
print(response.model_dump_json(indent=2))

# チェックポイント確認
response = client.fine_tuning.jobs.checkpoints.list(job_id)
print(response.model_dump_json(indent=2))

# モデル情報確認
response = client.fine_tuning.jobs.retrieve(job_id)
print(response.model_dump_json(indent=2))
fine_tuned_model = response.fine_tuned_model
```

```json: イベント
{
  "data": [
    {
      "id": "ftevent-f7b41a245cf24b8a8c0fa5f189ca54c9",
      "created_at": 1740447221,
      "level": "info",
      "message": "Training tokens billed: 18000",
      "object": "fine_tuning.job.event",
      "data": null,
      "type": "message"
    },
    {
      "id": "ftevent-28ff0c1de65f47a8b900f20073aaeccc",
      "created_at": 1740447221,
      "level": "info",
      "message": "Model Evaluation Passed.",
      "object": "fine_tuning.job.event",
      "data": null,
      "type": "message"
    },
    {
      "id": "ftevent-89a23c27377e4066b96e684d64408bf8",
      "created_at": 1740447221,
      "level": "info",
      "message": "Completed results file: file-3bf8c4ac95124fcf92947d63183685b4",
      "object": "fine_tuning.job.event",
      "data": null,
      "type": "message"
    },
    {
      "id": "ftevent-f3fb4f3718fb4e908a8c4868003ad603",
      "created_at": 1740447217,
      "level": "info",
      "message": "Postprocessing started.",
      "object": "fine_tuning.job.event",
      "data": null,
      "type": "message"
    },
    {
      "id": "ftevent-93570f3440b941968d81d717815b08c3",
      "created_at": 1740447172,
      "level": "info",
      "message": "Job succeeded.",
      "object": "fine_tuning.job.event",
      "data": null,
      "type": "message"
    },
    {
      "id": "ftevent-008dd553b129a1e008dd553b129a1e00",
      "created_at": 1740446636,
      "level": "info",
      "message": "Step 240: training loss=0.11896172910928726",
      "object": "fine_tuning.job.event",
      "data": {
        "step": 240,
        "train_loss": 0.11896172910928726,
        "train_mean_token_accuracy": 0.9583333134651184,
        "valid_loss": 0.7514890943254743,
        "valid_mean_token_accuracy": 0.75,
        "full_valid_loss": 0.7561612964140302,
        "full_valid_mean_token_accuracy": 0.7859922178988327
      },
      "type": "metrics"
    },
    {
      "id": "ftevent-008dd553b0ca43d008dd553b0ca43d00",
      "created_at": 1740446626,
      "level": "info",
      "message": "Step 230: training loss=0.20184214413166046",
      "object": "fine_tuning.job.event",
      "data": {
        "step": 230,
        "train_loss": 0.20184214413166046,
        "train_mean_token_accuracy": 0.9166666865348816,
        "valid_loss": 1.520263459947374,
        "valid_mean_token_accuracy": 0.5185185185185185
      },
      "type": "metrics"
    },
    {
      "id": "ftevent-008dd553b06ae5c008dd553b06ae5c00",
      "created_at": 1740446616,
      "level": "info",
      "message": "Step 220: training loss=0.4186785817146301",
      "object": "fine_tuning.job.event",
      "data": {
        "step": 220,
        "train_loss": 0.4186785817146301,
        "train_mean_token_accuracy": 0.875,
        "valid_loss": 0.7736611366271973,
        "valid_mean_token_accuracy": 0.7857142857142857
      },
      "type": "metrics"
    },
    {
      "id": "ftevent-008dd553b00b87b008dd553b00b87b00",
      "created_at": 1740446606,
      "level": "info",
      "message": "Step 210: training loss=0.29435476660728455",
      "object": "fine_tuning.job.event",
      "data": {
        "step": 210,
        "train_loss": 0.29435476660728455,
        "train_mean_token_accuracy": 0.875,
        "valid_loss": 1.528468096697772,
        "valid_mean_token_accuracy": 0.5555555555555556
      },
      "type": "metrics"
    },
    {
      "id": "ftevent-008dd553afac29a008dd553afac29a00",
      "created_at": 1740446596,
      "level": "info",
      "message": "Step 200: training loss=0.1939312368631363",
      "object": "fine_tuning.job.event",
      "data": {
        "step": 200,
        "train_loss": 0.1939312368631363,
        "train_mean_token_accuracy": 0.9545454382896423,
        "valid_loss": 0.8527921949114118,
        "valid_mean_token_accuracy": 0.8214285714285714
      },
      "type": "metrics"
    }
  ],
  "has_more": true,
  "object": "list"
}
```

```json: チェックポイント
{
  "data": [
    {
      "id": "ftchkpt-2c40f488cf7345609c73a8cadcabeb25",
      "created_at": 1740446850,
      "fine_tuned_model_checkpoint": "gpt-4o-mini-2024-07-18.ft-d78af615cda84674b541c026e764bcb7",
      "fine_tuning_job_id": "ftjob-d78af615cda84674b541c026e764bcb7",
      "metrics": {
        "full_valid_loss": 0.7561612964140302,
        "full_valid_mean_token_accuracy": 0.7859922178988327,
        "step": 240.0,
        "train_loss": 0.11896172910928726,
        "train_mean_token_accuracy": 0.9583333134651184,
        "valid_loss": 0.7514890943254743,
        "valid_mean_token_accuracy": 0.75
      },
      "object": "fine_tuning.job.checkpoint",
      "step_number": 240
    },
    {
      "id": "ftchkpt-18c4b5df9a69445aa6060921028f5fc0",
      "created_at": 1740446647,
      "fine_tuned_model_checkpoint": "gpt-4o-mini-2024-07-18.ft-d78af615cda84674b541c026e764bcb7:ckpt-step-160",
      "fine_tuning_job_id": "ftjob-d78af615cda84674b541c026e764bcb7",
      "metrics": {
        "full_valid_loss": 0.681790496588681,
        "full_valid_mean_token_accuracy": 0.7957198443579766,
        "step": 160.0,
        "train_loss": 0.25228747725486755,
        "train_mean_token_accuracy": 0.9629629850387573,
        "valid_loss": 0.807875428880964,
        "valid_mean_token_accuracy": 0.75
      },
      "object": "fine_tuning.job.checkpoint",
      "step_number": 160
    },
    {
      "id": "ftchkpt-6e5a718fc6db47958e2b7a195b318ae3",
      "created_at": 1740446522,
      "fine_tuned_model_checkpoint": "gpt-4o-mini-2024-07-18.ft-d78af615cda84674b541c026e764bcb7:ckpt-step-80",
      "fine_tuning_job_id": "ftjob-d78af615cda84674b541c026e764bcb7",
      "metrics": {
        "full_valid_loss": 0.6571730231496611,
        "full_valid_mean_token_accuracy": 0.7957198443579766,
        "step": 80.0,
        "train_loss": 0.6486645936965942,
        "train_mean_token_accuracy": 0.8888888955116272,
        "valid_loss": 0.8158819334847587,
        "valid_mean_token_accuracy": 0.75
      },
      "object": "fine_tuning.job.checkpoint",
      "step_number": 80
    }
  ],
  "has_more": false,
  "object": "list"
}
```

```json: モデル
{
  "id": "ftjob-d78af615cda84674b541c026e764bcb7",
  "created_at": 1740444480,
  "error": null,
  "fine_tuned_model": "gpt-4o-mini-2024-07-18.ft-d78af615cda84674b541c026e764bcb7",
  "finished_at": 1740447221,
  "hyperparameters": {
    "batch_size": 1,
    "learning_rate_multiplier": 1.0,
    "n_epochs": 3
  },
  "model": "gpt-4o-mini-2024-07-18",
  "object": "fine_tuning.job",
  "organization_id": null,
  "result_files": [
    "file-3bf8c4ac95124fcf92947d63183685b4"
  ],
  "seed": 105,
  "status": "succeeded",
  "trained_tokens": 12009,
  "training_file": "file-79600873356049e084973a13ea571a40",
  "validation_file": "file-637528dab95e418f97956a625a424960",
  "estimated_finish": 1740446395,
  "integrations": null,
  "method": null
}
```

#### 7. ファインチューニング済モデルデプロイ
`az account get-access-token`コマンドでTokenを取得し、コード内<token>に埋め込みます。
他にもリソースグループ、Azure OpenAI リソースの名前などをコード内に埋め込んでください。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/134273/95bd10db-6a34-47e6-890f-aa074612d404.png)

詳しくは以下を参照。

https://learn.microsoft.com/ja-jp/azure/ai-services/openai/tutorials/fine-tune?tabs=python-new%2Ccommand-line#deploy-fine-tuned-model

```python:
import requests

# Deploy fine-tuned model

token = <token>
subscription = "<Subscription ID>"
resource_group = "<Resource Group Name>"
resource_name = "<Azure Open AI Resource name>"
model_deployment_name = "gpt-4o-mini-2024-07-18-ft" # Custom deployment name you chose for your fine-tuning model

deploy_params = {'api-version': "2023-05-01"}
deploy_headers = {'Authorization': 'Bearer {}'.format(token), 'Content-Type': 'application/json'}

deploy_data = {
    "sku": {"name": "standard", "capacity": 1},
    "properties": {
        "model": {
            "format": "OpenAI",
            "name": fine_tuned_model, #retrieve this value from the previous call, it will look like gpt-4o-mini-2024-07-18.ft-0e208cf33a6a466994aff31a08aba678
            "version": "1"
        }
    }
}
deploy_data = json.dumps(deploy_data)

request_url = f'https://management.azure.com/subscriptions/{subscription}/resourceGroups/{resource_group}/providers/Microsoft.CognitiveServices/accounts/{resource_name}/deployments/{model_deployment_name}'

print('Creating a new deployment...')

r = requests.put(request_url, params=deploy_params, headers=deploy_headers, data=deploy_data)

print(r)
print(r.reason)
print(r.json())
```

#### 8. 結果確認 
Chatを投げて結果を確認します。

```python:
response = client.chat.completions.create(
    model = "gpt-4o-mini-2024-07-18-ft", # model = "Custom deployment name you chose for your fine-tuning model"
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "福原 洋平の趣味をいくつかあげて"},
    ]
)

print(response.choices[0].message.content)
```

ファインチューニングしたデータが反映されています。

```bash
福原 洋平の趣味には、登山、写真撮影、料理が含まれます。  
```