Hugging Face's logo

ce-lery
/
japanese-mistral-300m-instruction

Text Generation
Transformers
TensorBoard
Safetensors
mistral
Generated from Trainer
Inference Endpoints
text-generation-inference
Model card Files Metrics Community
japanese-mistral-300m-instruction
File size: 7,830 Bytes 
025d0db
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
 
---
base_model: ce-lery/japanese-mistral-300m-base
tags:
- generated_from_trainer
model-index:
- name: checkpoints-finetuning
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# japanese-mistral-300m-instruction

## Overview

Welcome to my model card!   

This Model feature is ...

- Suppression of unknown word generation by using byte fallback in SentencePiece tokenizer and conversion to huggingface Tokenizers format
- Pretrained by wikipedia dataset and cc100 dataset
- Use of [Mistral 300M](https://huggingface.co/ce-lery/japanese-mistral-300m-base/blob/main/config.json)
- Fine-tuning [ce-lery/japanese-mistral-300m-base](https://huggingface.co/ce-lery/japanese-mistral-300m-base) with [kunishou/databricks-dolly-15k-ja](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja)

Yukkuri shite ittene!

## How to use the model

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import os

MODEL_NAME = "ce-lery/japanese-mistral-300m-instruction"
torch.set_float32_matmul_precision('high')

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=False,trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME,trust_remote_code=True).to(device)

MAX_ASSISTANT_LENGTH = 100
MAX_INPUT_LENGTH = 128
INPUT_PROMPT = r'<s>\n以下は、タスクを説明する指示と、文脈のある入力の組み合わせです。要求を適切に満たす応答を書きなさい。\n[SEP]\n指示:\n{instruction}\n[SEP]\n入力:\n{input}\n[SEP]\n応答:\n'
NO_INPUT_PROMPT = r'<s>\n以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。\n[SEP]\n指示:\n{instruction}\n[SEP]\n応答:\n'

def prepare_input(instruction, input_text):
    if input_text != "":
        prompt = INPUT_PROMPT.format(instruction=instruction, input=input_text)
    else:
        prompt = NO_INPUT_PROMPT.format(instruction=instruction)
    return prompt

def format_output(output):
    output = output.lstrip("<s>").rstrip("</s>").replace("[SEP]", "").replace("\\n", "\n")
    return output

def generate_response(instruction, input_text):
    prompt = prepare_input(instruction, input_text)
    token_ids = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
    n = len(token_ids[0])
    # print(n)

    with torch.no_grad():
        output_ids = model.generate(
            token_ids.to(model.device),
            min_length=n,
            max_length=min(MAX_INPUT_LENGTH, n + MAX_ASSISTANT_LENGTH),
            top_p=0.95,
            top_k=50,
            temperature=0.4,
            do_sample=True,
            no_repeat_ngram_size=2,
            num_beams=3,
            pad_token_id=tokenizer.pad_token_id,
            bos_token_id=tokenizer.bos_token_id,
            eos_token_id=tokenizer.eos_token_id,
            bad_words_ids=[[tokenizer.unk_token_id]]
        )

    output = tokenizer.decode(output_ids.tolist()[0])
    formatted_output_all = format_output(output)
    response = f"Assistant:{formatted_output_all.split('応答:')[-1].strip()}"

    return formatted_output_all, response 

instruction = "あなたは何でも正確に答えられるAIです。"
questions = [
    "日本で一番高い山は？",
    "日本で一番広い湖は？",
    "世界で一番高い山は？",
    "世界で一番広い湖は？",
    "冗談を言ってください。",
]

# 各質問に対して応答を生成して表示
for question in questions:
    formatted_output_all, response = generate_response(instruction, question)
    print(response)

```

## Receipe

If you want to restruct this model, you can refer [this Github repository](https://github.com/ce-lery/japanese-mistral-300m-recipe).

I wrote the receipe for struction this model. For example,

- Preprocess with sentencepiece
- Pretraining with flash attention2 and torch.compile and DeepSpeed
- Fine-tuning with databricks-dolly-15k-ja

If you find my mistake,error,...etc, please create issue.
If you create pulreqest, I'm very happy! 
## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 64
- total_train_batch_size: 256
- optimizer: Adam with betas=(0.9,0.95) and epsilon=0.0001
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 1000
- num_epochs: 200
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch  | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 3.595         | 3.51   | 40   | 3.5299          |
| 3.4769        | 7.02   | 80   | 3.3722          |
| 3.3037        | 10.53  | 120  | 3.1871          |
| 3.1255        | 14.05  | 160  | 3.0088          |
| 2.9615        | 17.56  | 200  | 2.8684          |
| 2.8468        | 21.07  | 240  | 2.7808          |
| 2.7699        | 24.58  | 280  | 2.7205          |
| 2.7139        | 28.09  | 320  | 2.6793          |
| 2.6712        | 31.6   | 360  | 2.6509          |
| 2.6356        | 35.12  | 400  | 2.6294          |
| 2.6048        | 38.63  | 440  | 2.6120          |
| 2.5823        | 42.14  | 480  | 2.5974          |
| 2.5536        | 45.65  | 520  | 2.5849          |
| 2.5293        | 49.16  | 560  | 2.5740          |
| 2.5058        | 52.67  | 600  | 2.5644          |
| 2.482         | 56.19  | 640  | 2.5556          |
| 2.4575        | 59.7   | 680  | 2.5477          |
| 2.4339        | 63.21  | 720  | 2.5405          |
| 2.4073        | 66.72  | 760  | 2.5350          |
| 2.3845        | 70.23  | 800  | 2.5303          |
| 2.3606        | 73.74  | 840  | 2.5253          |
| 2.329         | 77.26  | 880  | 2.5215          |
| 2.3071        | 80.77  | 920  | 2.5185          |
| 2.2768        | 84.28  | 960  | 2.5155          |
| 2.2479        | 87.79  | 1000 | 2.5144          |
| 2.2181        | 91.3   | 1040 | 2.5151          |
| 2.1901        | 94.81  | 1080 | 2.5139          |
| 2.1571        | 98.33  | 1120 | 2.5148          |
| 2.1308        | 101.84 | 1160 | 2.5166          |
| 2.1032        | 105.35 | 1200 | 2.5193          |
| 2.0761        | 108.86 | 1240 | 2.5204          |
| 2.0495        | 112.37 | 1280 | 2.5269          |
| 2.0231        | 115.88 | 1320 | 2.5285          |
| 2.0021        | 119.4  | 1360 | 2.5328          |
| 1.9793        | 122.91 | 1400 | 2.5383          |
| 1.9575        | 126.42 | 1440 | 2.5442          |
| 1.9368        | 129.93 | 1480 | 2.5488          |
| 1.9216        | 133.44 | 1520 | 2.5534          |
| 1.902         | 136.95 | 1560 | 2.5584          |
| 1.8885        | 140.47 | 1600 | 2.5609          |
| 1.8728        | 143.98 | 1640 | 2.5657          |
| 1.8605        | 147.49 | 1680 | 2.5697          |
| 1.8476        | 151.0  | 1720 | 2.5741          |
| 1.8402        | 154.51 | 1760 | 2.5770          |
| 1.8274        | 158.02 | 1800 | 2.5803          |
| 1.8218        | 161.54 | 1840 | 2.5829          |
| 1.8144        | 165.05 | 1880 | 2.5847          |
| 1.8097        | 168.56 | 1920 | 2.5867          |
| 1.8076        | 172.07 | 1960 | 2.5883          |
| 1.8014        | 175.58 | 2000 | 2.5892          |
| 1.8001        | 179.09 | 2040 | 2.5899          |
| 1.7987        | 182.61 | 2080 | 2.5903          |
| 1.7971        | 186.12 | 2120 | 2.5906          |
| 1.7979        | 189.63 | 2160 | 2.5907          |
| 1.7975        | 193.14 | 2200 | 2.5907          |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.1+cu121
- Datasets 2.14.5
- Tokenizers 0.14.1