feat: fine-tuning japanese-mistral-300M-base with databricks-dolly-15k-ja

025d0db 11 months ago

7.83 kB

	---
	base_model: ce-lery/japanese-mistral-300m-base
	tags:
	- generated_from_trainer
	model-index:
	- name: checkpoints-finetuning
	results: []
	---

	<!-- This model card has been generated automatically according to the information the Trainer had access to. You
	should probably proofread and complete it, then remove this comment. -->

	# japanese-mistral-300m-instruction

	## Overview

	Welcome to my model card!

	This Model feature is ...

	- Suppression of unknown word generation by using byte fallback in SentencePiece tokenizer and conversion to huggingface Tokenizers format
	- Pretrained by wikipedia dataset and cc100 dataset
	- Use of [Mistral 300M](https://huggingface.co/ce-lery/japanese-mistral-300m-base/blob/main/config.json)
	- Fine-tuning [ce-lery/japanese-mistral-300m-base](https://huggingface.co/ce-lery/japanese-mistral-300m-base) with [kunishou/databricks-dolly-15k-ja](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja)

	Yukkuri shite ittene!

	## How to use the model

	```python
	import torch
	from transformers import AutoTokenizer, AutoModelForCausalLM
	import os

	MODEL_NAME = "ce-lery/japanese-mistral-300m-instruction"
	torch.set_float32_matmul_precision('high')

	device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
	print(device)

	tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=False,trust_remote_code=True)
	model = AutoModelForCausalLM.from_pretrained(MODEL_NAME,trust_remote_code=True).to(device)

	MAX_ASSISTANT_LENGTH = 100
	MAX_INPUT_LENGTH = 128
	INPUT_PROMPT = r'<s>\n以下は、タスクを説明する指示と、文脈のある入力の組み合わせです。要求を適切に満たす応答を書きなさい。\n[SEP]\n指示:\n{instruction}\n[SEP]\n入力:\n{input}\n[SEP]\n応答:\n'
	NO_INPUT_PROMPT = r'<s>\n以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。\n[SEP]\n指示:\n{instruction}\n[SEP]\n応答:\n'

	def prepare_input(instruction, input_text):
	if input_text != "":
	prompt = INPUT_PROMPT.format(instruction=instruction, input=input_text)
	else:
	prompt = NO_INPUT_PROMPT.format(instruction=instruction)
	return prompt

	def format_output(output):
	output = output.lstrip("<s>").rstrip("</s>").replace("[SEP]", "").replace("\\n", "\n")
	return output

	def generate_response(instruction, input_text):
	prompt = prepare_input(instruction, input_text)
	token_ids = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
	n = len(token_ids[0])
	# print(n)

	with torch.no_grad():
	output_ids = model.generate(
	token_ids.to(model.device),
	min_length=n,
	max_length=min(MAX_INPUT_LENGTH, n + MAX_ASSISTANT_LENGTH),
	top_p=0.95,
	top_k=50,
	temperature=0.4,
	do_sample=True,
	no_repeat_ngram_size=2,
	num_beams=3,
	pad_token_id=tokenizer.pad_token_id,
	bos_token_id=tokenizer.bos_token_id,
	eos_token_id=tokenizer.eos_token_id,
	bad_words_ids=[[tokenizer.unk_token_id]]
	)

	output = tokenizer.decode(output_ids.tolist()[0])
	formatted_output_all = format_output(output)
	response = f"Assistant:{formatted_output_all.split('応答:')[-1].strip()}"

	return formatted_output_all, response

	instruction = "あなたは何でも正確に答えられるAIです。"
	questions = [
	"日本で一番高い山は？",
	"日本で一番広い湖は？",
	"世界で一番高い山は？",
	"世界で一番広い湖は？",
	"冗談を言ってください。",
	]

	# 各質問に対して応答を生成して表示
	for question in questions:
	formatted_output_all, response = generate_response(instruction, question)
	print(response)

	```

	## Receipe

	If you want to restruct this model, you can refer [this Github repository](https://github.com/ce-lery/japanese-mistral-300m-recipe).

	I wrote the receipe for struction this model. For example,

	- Preprocess with sentencepiece
	- Pretraining with flash attention2 and torch.compile and DeepSpeed
	- Fine-tuning with databricks-dolly-15k-ja

	If you find my mistake,error,...etc, please create issue.
	If you create pulreqest, I'm very happy!
	## Training procedure

	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 5e-06
	- train_batch_size: 4
	- eval_batch_size: 4
	- seed: 42
	- distributed_type: multi-GPU
	- gradient_accumulation_steps: 64
	- total_train_batch_size: 256
	- optimizer: Adam with betas=(0.9,0.95) and epsilon=0.0001
	- lr_scheduler_type: cosine
	- lr_scheduler_warmup_steps: 1000
	- num_epochs: 200
	- mixed_precision_training: Native AMP

	### Training results

	\| Training Loss \| Epoch \| Step \| Validation Loss \|
	\|:-------------:\|:------:\|:----:\|:---------------:\|
	\| 3.595 \| 3.51 \| 40 \| 3.5299 \|
	\| 3.4769 \| 7.02 \| 80 \| 3.3722 \|
	\| 3.3037 \| 10.53 \| 120 \| 3.1871 \|
	\| 3.1255 \| 14.05 \| 160 \| 3.0088 \|
	\| 2.9615 \| 17.56 \| 200 \| 2.8684 \|
	\| 2.8468 \| 21.07 \| 240 \| 2.7808 \|
	\| 2.7699 \| 24.58 \| 280 \| 2.7205 \|
	\| 2.7139 \| 28.09 \| 320 \| 2.6793 \|
	\| 2.6712 \| 31.6 \| 360 \| 2.6509 \|
	\| 2.6356 \| 35.12 \| 400 \| 2.6294 \|
	\| 2.6048 \| 38.63 \| 440 \| 2.6120 \|
	\| 2.5823 \| 42.14 \| 480 \| 2.5974 \|
	\| 2.5536 \| 45.65 \| 520 \| 2.5849 \|
	\| 2.5293 \| 49.16 \| 560 \| 2.5740 \|
	\| 2.5058 \| 52.67 \| 600 \| 2.5644 \|
	\| 2.482 \| 56.19 \| 640 \| 2.5556 \|
	\| 2.4575 \| 59.7 \| 680 \| 2.5477 \|
	\| 2.4339 \| 63.21 \| 720 \| 2.5405 \|
	\| 2.4073 \| 66.72 \| 760 \| 2.5350 \|
	\| 2.3845 \| 70.23 \| 800 \| 2.5303 \|
	\| 2.3606 \| 73.74 \| 840 \| 2.5253 \|
	\| 2.329 \| 77.26 \| 880 \| 2.5215 \|
	\| 2.3071 \| 80.77 \| 920 \| 2.5185 \|
	\| 2.2768 \| 84.28 \| 960 \| 2.5155 \|
	\| 2.2479 \| 87.79 \| 1000 \| 2.5144 \|
	\| 2.2181 \| 91.3 \| 1040 \| 2.5151 \|
	\| 2.1901 \| 94.81 \| 1080 \| 2.5139 \|
	\| 2.1571 \| 98.33 \| 1120 \| 2.5148 \|
	\| 2.1308 \| 101.84 \| 1160 \| 2.5166 \|
	\| 2.1032 \| 105.35 \| 1200 \| 2.5193 \|
	\| 2.0761 \| 108.86 \| 1240 \| 2.5204 \|
	\| 2.0495 \| 112.37 \| 1280 \| 2.5269 \|
	\| 2.0231 \| 115.88 \| 1320 \| 2.5285 \|
	\| 2.0021 \| 119.4 \| 1360 \| 2.5328 \|
	\| 1.9793 \| 122.91 \| 1400 \| 2.5383 \|
	\| 1.9575 \| 126.42 \| 1440 \| 2.5442 \|
	\| 1.9368 \| 129.93 \| 1480 \| 2.5488 \|
	\| 1.9216 \| 133.44 \| 1520 \| 2.5534 \|
	\| 1.902 \| 136.95 \| 1560 \| 2.5584 \|
	\| 1.8885 \| 140.47 \| 1600 \| 2.5609 \|
	\| 1.8728 \| 143.98 \| 1640 \| 2.5657 \|
	\| 1.8605 \| 147.49 \| 1680 \| 2.5697 \|
	\| 1.8476 \| 151.0 \| 1720 \| 2.5741 \|
	\| 1.8402 \| 154.51 \| 1760 \| 2.5770 \|
	\| 1.8274 \| 158.02 \| 1800 \| 2.5803 \|
	\| 1.8218 \| 161.54 \| 1840 \| 2.5829 \|
	\| 1.8144 \| 165.05 \| 1880 \| 2.5847 \|
	\| 1.8097 \| 168.56 \| 1920 \| 2.5867 \|
	\| 1.8076 \| 172.07 \| 1960 \| 2.5883 \|
	\| 1.8014 \| 175.58 \| 2000 \| 2.5892 \|
	\| 1.8001 \| 179.09 \| 2040 \| 2.5899 \|
	\| 1.7987 \| 182.61 \| 2080 \| 2.5903 \|
	\| 1.7971 \| 186.12 \| 2120 \| 2.5906 \|
	\| 1.7979 \| 189.63 \| 2160 \| 2.5907 \|
	\| 1.7975 \| 193.14 \| 2200 \| 2.5907 \|


	### Framework versions

	- Transformers 4.35.2
	- Pytorch 2.1.1+cu121
	- Datasets 2.14.5
	- Tokenizers 0.14.1