File size: 2,184 Bytes
51d7b1b 49f6872 51d7b1b 166e592 51d7b1b 06a55a1 51d7b1b 166e592 51d7b1b 49f6872 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 |
---
license: other
license_name: gemma-terms-of-use
license_link: https://ai.google.dev/gemma/terms
language:
- zh
pipeline_tag: text-generation
datasets: linux-cn/archive
tags:
- unsloth
---
# 介绍
本模型主要用途为基于科技类文章生成对应标题。
本次将开源从 100-2200 steps 的中间所有 checkpoint 以供大家参考。
# 使用
```python
from transformers import AutoModelForCausalLM, AutoTokenizer,BitsAndBytesConfig
peft_model_id = "checkpoint-2000"
model = AutoModelForCausalLM.from_pretrained(peft_model_id,device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
input_text = """
Generate a title for the article:
{content}
---
Title:
""" # 固定格式
encoding = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**encoding,max_length=8192,temperature=0.2,do_sample=True)
generated_ids = outputs[:, encoding.input_ids.shape[1]:]
generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_texts[0])
```
# 训练数据
linux-cn 文章
https://huggingface.co/datasets/linux-cn/archive
# 微调
基于 [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) 在单张A100（80G）上进行微调，微调参数如下：
```bash
CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
--stage sft \
--do_train True \
--model_name_or_path google/gemma-2b \
--finetuning_type lora \
--template default \
--dataset title \
--use_unsloth \
--cutoff_len 8192 \
--learning_rate 5e-05 \
--num_train_epochs 10.0 \
--max_samples 10000 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--gradient_accumulation_steps 4 \
--lr_scheduler_type cosine \
--max_grad_norm 1.0 \
--logging_steps 10 \
--save_steps 100 \
--eval_steps 100 \
--evaluation_strategy steps \
--warmup_steps 0 \
--output_dir saves/Gemma-2B/lora/train_2024-03-01-04-36-32 \
--bf16 True \
--lora_rank 8 \
--lora_dropout 0.1 \
--lora_target q_proj,v_proj \
--val_size 0.1 \
--load_best_model_at_end True \
--plot_loss True \
--report_to "tensorboard"
``` |