BEE-spoke-data/smol_llama-220M-bees-internal-GGUF

Quantized GGUF model files for smol_llama-220M-bees-internal from BEE-spoke-data

Name	Quant method	Size
smol_llama-220m-bees-internal.fp16.gguf	fp16	436.50 MB
smol_llama-220m-bees-internal.q2_k.gguf	q2_k	94.43 MB
smol_llama-220m-bees-internal.q3_k_m.gguf	q3_k_m	114.65 MB
smol_llama-220m-bees-internal.q4_k_m.gguf	q4_k_m	137.58 MB
smol_llama-220m-bees-internal.q5_k_m.gguf	q5_k_m	157.91 MB
smol_llama-220m-bees-internal.q6_k.gguf	q6_k	179.52 MB
smol_llama-220m-bees-internal.q8_0.gguf	q8_0	232.28 MB

Original Model Card:

smol_llama-220M-bees-internal

This model is a fine-tuned version of BEE-spoke-data/smol_llama-220M-GQA on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.6892
• Accuracy: 0.4610

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 2
• seed: 27634
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.0959	0.1	50	2.9671	0.4245
2.9975	0.19	100	2.8691	0.4371
2.8938	0.29	150	2.8271	0.4419
2.9027	0.39	200	2.7973	0.4457
2.8983	0.49	250	2.7719	0.4489
2.8789	0.58	300	2.7519	0.4515
2.8672	0.68	350	2.7366	0.4535
2.8369	0.78	400	2.7230	0.4558
2.8271	0.88	450	2.7118	0.4569
2.7775	0.97	500	2.7034	0.4587
2.671	1.07	550	2.6996	0.4592
2.695	1.17	600	2.6965	0.4598
2.6962	1.27	650	2.6934	0.4601
2.6034	1.36	700	2.6916	0.4605
2.716	1.46	750	2.6901	0.4609
2.6968	1.56	800	2.6896	0.4608
2.6626	1.66	850	2.6893	0.4609
2.6881	1.75	900	2.6891	0.4610
2.7339	1.85	950	2.6891	0.4610
2.6729	1.95	1000	2.6892	0.4610

Framework versions

• Transformers 4.36.2
• Pytorch 2.1.0
• Datasets 2.16.1
• Tokenizers 0.15.0