datasets: | |
- gozfarb/ShareGPT_Vicuna_unfiltered | |
# Convert tools | |
https://github.com/practicaldreamer/vicuna_to_alpaca | |
# Training tool | |
https://github.com/oobabooga/text-generation-webui | |
ATM I'm using 2023.05.04v0 of the dataset and training full context. | |
# Notes: | |
So I will only be training 1 epoch, as full context 30b takes so long to train. | |
This 1 epoch will take me 8 days lol but luckily these LoRA feels fully functinal at epoch 1 as shown on my 13b one. | |
Also I will be uploading checkpoints almost everyday. I could train another epoch if there's enough want for it. | |
# How to test? | |
1. Download LLaMA-30B-HF if you have not: https://huggingface.co/Neko-Institute-of-Science/LLaMA-30B-HF | |
2. Download the checkpoint-xxxx folder you want and put it in the loras folder. | |
3. Load ooba: ```python server.py --listen --model LLaMA-30B-HF --load-in-8bit --chat --lora checkpoint-xxxx``` | |
4. Select instruct and chose Vicuna-v1.1 template. | |
# Want to see it Training? | |
https://wandb.ai/neko-science/VicUnLocked/runs/vx8yzwi7 |