軽量LLMでCPU環境でプロンプトエンジニアリングを試す

LLMを動かしながらプロンプトエンジニアリングを試してみました。超軽量の[flan-t5-base](https://huggingface.co/google/flan-t5-base)を使って文章要約タスクを試しています。パラメータ数が248Mと少なく、手軽に動かすには都合が良かったです。その反面日本語ではプロンプトできないですし、精度も良くないです。Google ColabのCPUで動かしています。

# 実行環境
2024/3/26時点のGoogle Colab で、Python3.10.12で、以下のパッケージを使っています。

|Package|Version|備考|
|:-:|:-:|:--|
|torch|2.1.0+cu121|Google Colabにプリインストールされていたバージョン|
|transformers|4.38.2|Google Colabにプリインストールされていたバージョン|
|datasets|2.18.0|追加でpip install|

# サマリ
以下のダイアログ要約をパターン変えてさせています。

```text: knkarthick/dialogsum test-11
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
```

## 正解ラベルと各パターン別結果
正解ラベルと、各試行パターンごとの生成値です(tempertureはdefault1)。
モデルがあんまり良くないので0 shotと 1 shotで違いが微妙だし、2shotでも大して良くない。
tempertureを2にすると滅茶苦茶になりますね。

|種別|結果|
|:--|:--|
|正解ラベル|#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
|0 shot without instruction|Brian, thank you for coming to our party.||
|0 shot|Brian's birthday is coming up.||
|1 shot|Brian's birthday is coming up.||
|2 shot(temperture=1)|Brian's birthday is coming up. Brian wants to have a dance with Brian.||
|2 shot(temperture=0.1)|Brian's birthday is coming up. Brian will have a dance with Brian.||
|2 shot(temperture=0.5)|Brian's birthday is coming up. Brian invites everyone to come over and have a dance.|
|2 shot(temperture=2)|#p and Others enjoy and love a private dance, with dancing of celebrities' ward.|

# 実行プログラム
## 1. pip intall
Google Colab にプレインストールされていないdatasetsをインストール。

```terminal
!pip install datasets
```

## 2. Impot packages
PythonパッケージをImport。`AutoModelForSeq2SeqLM`(Encoder-Decoder)は珍しく、GPTなどDecoder-onlyが主流(その場合は`AutoModelForCausalLM`)。

```python
rom datasets import load_dataset

# AutoModelForSeq2SeqLM は Encoder-decoder用。 AutoModelForCausalLM だと auto-regressive(Decoder-only)
from transformers import AutoModelForSeq2SeqLM
from transformers import AutoTokenizer
from transformers import GenerationConfig
```

## 3. Load Data 
以下のdialogsumのデータを使っています。ダイアログと要約のデータがあります。

https://huggingface.co/datasets/knkarthick/dialogsum

データをロードします。

```python
dataset = load_dataset('knkarthick/dialogsum')
print(dataset)
```

```text: 出力結果
DatasetDict({
    train: Dataset({
        features: ['id', 'dialogue', 'summary', 'topic'],
        num_rows: 12460
    })
    validation: Dataset({
        features: ['id', 'dialogue', 'summary', 'topic'],
        num_rows: 500
    })
    test: Dataset({
        features: ['id', 'dialogue', 'summary', 'topic'],
        num_rows: 1500
    })
})
```

データの中身を見てみます。

```python
# index 1, 2は同じダイアログだったので離れたIndexにした
examples = [1, 10]
dash_line = '-'*100

def output_prompt(dataset, i, prompt):
    print(dash_line)
    print('例: ', i + 1)
    print(dash_line)
    print('プロンプト:')
    print(prompt)
    print(dash_line)

def output_result(y_true, output=None):
    print('正解ラベル:')
    print(y_true)
    print(dash_line)
    if output != None:
        print(f'モデル生成結果:\n{output}\n')
    print()

for i in examples:
    output_prompt(dataset, i, dataset['test'][i]['dialogue'])
    output_result(dataset['test'][i]['summary'])
```

```text: 出力結果
----------------------------------------------------------------------------------------------------
例:  2
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.
----------------------------------------------------------------------------------------------------
正解ラベル:
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
----------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------
例:  11
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
```

## 4. Load model
[Flan-T5 release](https://huggingface.co/collections/google/flan-t5-release-65005c39e3201fff885e22fb)のbaseサイズを使いました。パラメータ数は248Mで昨今のLLMに比べると非常に少ないです。

```python
# https://huggingface.co/google/flan-t5-small
# use Encoder-decoder small model
#model_name='google/flan-t5-small'
model_name='google/flan-t5-base'
#model_name='google/flan-t5-large'
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
```

Tokenizerを少し試してみます。

```python
# 日本語で試したが当然、Encodeできない
sentence = "What time is it, Tom?"

# PyTorchのテンソル型で返すためにreturn_tensors='pt'
sentence_encoded = tokenizer(sentence, return_tensors='pt')

sentence_decoded = tokenizer.decode(
        sentence_encoded["input_ids"][0],
        skip_special_tokens=True
    )

print(f'{tokenizer.max_model_input_sizes=}')

print('ENCODED SENTENCE:')
print(sentence_encoded["input_ids"][0])

print('\nENCODED Tokens:')
print(tokenizer.convert_ids_to_tokens(sentence_encoded["input_ids"][0]))

print('\nDECODED SENTENCE:')
print(sentence_decoded)
```


```text: 出力結果
tokenizer.max_model_input_sizes={'google-t5/t5-small': 512, 'google-t5/t5-base': 512, 'google-t5/t5-large': 512, 'google-t5/t5-3b': 512, 'google-t5/t5-11b': 512}
ENCODED SENTENCE:
tensor([ 363,   97,   19,   34,    6, 3059,   58,    1])

ENCODED Tokens:
['▁What', '▁time', '▁is', '▁it', ',', '▁Tom', '?', '
']

DECODED SENTENCE:
What time is it, Tom?
```

## 5. Prompt Engineering
ここから本題です。
事前にサマリを生成する関数を定義しておきます。

```python
def generate_summary(tokenizer, prompt, config=GenerationConfig(max_new_tokens=50)):
    inputs = tokenizer(prompt, return_tensors='pt')
    output = tokenizer.decode(
        model.generate(
            inputs["input_ids"],
            max_new_tokens=50,
            generation_config=config
        )[0],
        skip_special_tokens=True
    )
    return output
```

### 5.1. Zero-Shot without instruction
Zero-ShotでインストラクションなしでLLMに投げます。

```python
for i in examples:
    dialogue = dataset['test'][i]['dialogue']
    output = generate_summary(tokenizer, dialogue)
    output_prompt(dataset, i, dialogue)
    output_result(dataset['test'][i]['summary'], output)
```

```text: 出力結果
----------------------------------------------------------------------------------------------------
例:  2
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.
----------------------------------------------------------------------------------------------------
正解ラベル:
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
----------------------------------------------------------------------------------------------------
モデル生成結果:
#Person1#: Ms. Dawson, I need you to take a dictation for me.


----------------------------------------------------------------------------------------------------
例:  11
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian, thank you for coming to our party.
```

### 5.2. Zero-Shot with instruction
Zero-Shotでインストラクションとして「What was going on?」を付加してみます。もう少しいいインストラクションあるかもしれませんが、試していません。

```python
for i in examples:
    dialogue = dataset['test'][i]['dialogue']

    prompt = f"""
Dialogue:

{dialogue}

What was going on?
"""

    output = generate_summary(tokenizer, prompt)
    output_prompt(dataset, i, prompt)
    output_result(dataset['test'][i]['summary'], output)
```

```text: 出力結果
----------------------------------------------------------------------------------------------------
例:  2
----------------------------------------------------------------------------------------------------
Dialogue:

#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.

What was going on?

----------------------------------------------------------------------------------------------------
正解ラベル:
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
----------------------------------------------------------------------------------------------------
モデル生成結果:
The memo is to be distributed to all employees by this afternoon.


----------------------------------------------------------------------------------------------------
例:  11
----------------------------------------------------------------------------------------------------
プロンプト:

Dialogue:

#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday

What was going on?

----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up.
```

長いので、Instruction有無でどう変わったか改めて表にします。少し良くなっていますね。

|例|Without Instruction|With Instruction
|:-:|:--|:--|
|2|#Person1#: Ms. Dawson, I need you to take a dictation for me.|The memo is to be distributed to all employees by this afternoon.
|11|Brian, thank you for coming to our party.|Brian's birthday is coming up.


### 5.3. One-Shot  
次はOne-Shotで試してみます。

```python
def make_prompt(example_indices_full, example_index_to_summarize):
    prompt = ''
    for index in example_indices_full:

        # '{summary}の後の\n\n\n'がFLAN-T5にとっては重要らしい
        prompt += f"""
Dialogue:

{dataset['test'][index]['dialogue']}

What was going on?
{dataset['test'][index]['summary']}


"""

    prompt += f"""
Dialogue:

{dataset['test'][example_index_to_summarize]['dialogue']}

What was going on?
"""

    return prompt

output = generate_summary(tokenizer, one_shot_prompt)
output_result(dataset['test'][examples[1]]['summary'], output)
```

プロンプトの中身です。

```text: Prompt(One-Shot)
Dialogue:

#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.

What was going on?
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.



Dialogue:

#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday

What was going on?
```

ZeroーShotと変わらなかったです。

```text: 出力結果
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up.
```

### 5.４. Few-Shot  
Few-Shot(2 Shot)で試してみます。

```python
few_shot_prompt = make_prompt([examples[0], 80], examples[1])
print(few_shot_prompt)
```

Promptの内容です。

```text: Prompt(Few-Shot)
Dialogue:

#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.

What was going on?
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.



Dialogue:

#Person1#: May, do you mind helping me prepare for the picnic?
#Person2#: Sure. Have you checked the weather report?
#Person1#: Yes. It says it will be sunny all day. No sign of rain at all. This is your father's favorite sausage. Sandwiches for you and Daniel.
#Person2#: No, thanks Mom. I'd like some toast and chicken wings.
#Person1#: Okay. Please take some fruit salad and crackers for me.
#Person2#: Done. Oh, don't forget to take napkins disposable plates, cups and picnic blanket.
#Person1#: All set. May, can you help me take all these things to the living room?
#Person2#: Yes, madam.
#Person1#: Ask Daniel to give you a hand?
#Person2#: No, mom, I can manage it by myself. His help just causes more trouble.

What was going on?
Mom asks May to help to prepare for the picnic and May agrees.



Dialogue:

#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday

What was going on?
```

生成結果です。BrianはBrianと踊りたい、というのは意味がわからないですね。

```text: 出力結果
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. Brian wants to have a dance with Brian.
```

### 5.5. Configuration  
Configurationで`temperature`を変えてFew-Shotでやってみます。`temperature`が高いほどランダム性が高くなります。

```python
# temperature が高いとランダム性が高い(Softmaxレイヤの計算制御)
for temperature in [0.1, 0.5, 1.0, 2.0]:
    generation_config = GenerationConfig(max_new_tokens=50,
                                         do_sample=True, temperature=temperature)
    print(dash_line)
    print(f'{temperature=}')
    print(dash_line)
    output = generate_summary(tokenizer, few_shot_prompt, generation_config)
    output_result(dataset['test'][examples[1]]['summary'], output)
```

```text: 出力結果
----------------------------------------------------------------------------------------------------
temperature=0.1
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. Brian will have a dance with Brian.


----------------------------------------------------------------------------------------------------
temperature=0.5
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. Brian invites everyone to come over and have a dance.


----------------------------------------------------------------------------------------------------
temperature=1.0
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. People congratulate him on his appearance.


----------------------------------------------------------------------------------------------------
temperature=2.0
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
#p and Others enjoy and love a private dance, with dancing of celebrities' ward.
```