language:
- en
tags:
- code
About
Hi, this is the Readme.
This Model was created as a study experiment, to re-create alpaca on my end.
It uses the gururise/AlpacaDataCleaned Dataset ( From April 7 )
Specifications
Base Model:
ββLLaMA 7B
Training Parameters:
ββMicro_Batch_Size = 8
ββBatch_Size = 128
ββGradient_Accumulation_Steps = Batch_Size / Micro_Batch_Sizeβββ# ( 0.0625 )
ββEpochs = 2
ββLearning_Rate = 2e-5
ββCutoff_Len = 256βββ# This ( 256 ) accounts for about 96% of all data
ββLora_R = 4
ββLora_Alpha = 16
ββLora_Dropout = 0.05
β β
Files
ββadapter_model.binβ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
# This is the Fine-tuned Weights that goes over the base LLaMA Model.
ββadapter_config.binβ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
# This is Config File for the adapter_model file.
ββconsolidated.00.pthβ
β
β
β
β
β
β
β
β
β
β
β
β
β
# This File is the Base Model File ( LLaMA 7B ), merged with the fine-tuned weights ( adapter_model.bin ).
ββtokenizer.modelβ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
# This is the tokenizer file, it converts the input text ( prompt ) to tokens that the NN can understand.
ββparams.jsonβ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
# Parameters of the Model.
ββggml_model_f16.binβ
β
β
β
β
β
β
β
β
β
β
β
# This is the same model ( consolidated.00.pth ), but now it's in 'ggml f16' format. We need this format to quantize it with llama.cpp.
ββllama-hf-7bβ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
# This folder contains the same model ( consolidated.00.pth ), but now it's in 'huggingface' format. We need this format to quantize it with GPTQ.
ββquantized-model:
ββββggml-model-q4_0.binβ
β
β
# This is the 4-bit Quantized Model by llama.cpp, I found this to be better than GPTQ.
ββββllama7b-4bit-128g.ptβ
β
β
β
# This is the Quantized Model by GPTQ. It takes longer to train and gives worse results compared to llama.cpp, but it does have a ( 7.6% ) smaller file size.