Hugging Face's logo

afrideva
/
verysmol_llama-v11-KIx2-GGUF

Text Generation
GGUF
Generated from Trainer
ggml
quantized
q2_k
q3_k_m
q4_k_m
q5_k_m
q6_k
q8_0
Model card Files Community
verysmol_llama-v11-KIx2-GGUF
File size: 6,980 Bytes 
edb90e0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
 
---
base_model: BEE-spoke-data/verysmol_llama-v11-KIx2
datasets:
- BEE-spoke-data/knowledge-inoc-concat-v1
inference: false
license: apache-2.0
metrics:
- accuracy
model_creator: BEE-spoke-data
model_name: verysmol_llama-v11-KIx2
pipeline_tag: text-generation
quantized_by: afrideva
tags:
- generated_from_trainer
- gguf
- ggml
- quantized
- q2_k
- q3_k_m
- q4_k_m
- q5_k_m
- q6_k
- q8_0
widget:
- example_title: El Microondas
  text: My name is El Microondas the Wise and
- example_title: Kennesaw State University
  text: Kennesaw State University is a public
- example_title: Bungie
  text: Bungie Studios is an American video game developer. They are most famous for
    developing the award winning Halo series of video games. They also made Destiny.
    The studio was founded
- example_title: Mona Lisa
  text: The Mona Lisa is a world-renowned painting created by
- example_title: Harry Potter Series
  text: The Harry Potter series, written by J.K. Rowling, begins with the book titled
- example_title: Riddle
  text: 'Question: I have cities, but no houses. I have mountains, but no trees. I
    have water, but no fish. What am I?

    Answer:'
- example_title: Photosynthesis
  text: The process of photosynthesis involves the conversion of
- example_title: Story Continuation
  text: Jane went to the store to buy some groceries. She picked up apples, oranges,
    and a loaf of bread. When she got home, she realized she forgot
- example_title: Math Problem
  text: 'Problem 2: If a train leaves Station A at 9:00 AM and travels at 60 mph,
    and another train leaves Station B at 10:00 AM and travels at 80 mph, when will
    they meet if the distance between the stations is 300 miles?

    To determine'
- example_title: Algorithm Definition
  text: In the context of computer programming, an algorithm is
---
# BEE-spoke-data/verysmol_llama-v11-KIx2-GGUF

Quantized GGUF model files for [verysmol_llama-v11-KIx2](https://huggingface.co/BEE-spoke-data/verysmol_llama-v11-KIx2) from [BEE-spoke-data](https://huggingface.co/BEE-spoke-data)


| Name | Quant method | Size |
| ---- | ---- | ---- |
| [verysmol_llama-v11-kix2.fp16.gguf](https://huggingface.co/afrideva/verysmol_llama-v11-KIx2-GGUF/resolve/main/verysmol_llama-v11-kix2.fp16.gguf) | fp16 | 116.89 MB  |
| [verysmol_llama-v11-kix2.q2_k.gguf](https://huggingface.co/afrideva/verysmol_llama-v11-KIx2-GGUF/resolve/main/verysmol_llama-v11-kix2.q2_k.gguf) | q2_k | 30.14 MB  |
| [verysmol_llama-v11-kix2.q3_k_m.gguf](https://huggingface.co/afrideva/verysmol_llama-v11-KIx2-GGUF/resolve/main/verysmol_llama-v11-kix2.q3_k_m.gguf) | q3_k_m | 33.71 MB  |
| [verysmol_llama-v11-kix2.q4_k_m.gguf](https://huggingface.co/afrideva/verysmol_llama-v11-KIx2-GGUF/resolve/main/verysmol_llama-v11-kix2.q4_k_m.gguf) | q4_k_m | 38.34 MB  |
| [verysmol_llama-v11-kix2.q5_k_m.gguf](https://huggingface.co/afrideva/verysmol_llama-v11-KIx2-GGUF/resolve/main/verysmol_llama-v11-kix2.q5_k_m.gguf) | q5_k_m | 43.21 MB  |
| [verysmol_llama-v11-kix2.q6_k.gguf](https://huggingface.co/afrideva/verysmol_llama-v11-KIx2-GGUF/resolve/main/verysmol_llama-v11-kix2.q6_k.gguf) | q6_k | 48.39 MB  |
| [verysmol_llama-v11-kix2.q8_0.gguf](https://huggingface.co/afrideva/verysmol_llama-v11-KIx2-GGUF/resolve/main/verysmol_llama-v11-kix2.q8_0.gguf) | q8_0 | 62.45 MB  |



## Original Model Card:
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# verysmol_llama-v11-KIx2

## Model description

This model is a fine-tuned version of v10 (refinedweb-3m dedup) further trained for 2 epochs on KI dataset.

It achieves the following results on the evaluation set:
- Loss: 2.8876
- Accuracy: 0.4502

---

## evals

`hf-causal-experimental (pretrained=pszemraj/verysmol_llama-v11-KIx2,revision=main,trust_remote_code=True,dtype='float'), limit: None, provide_description: False, num_fewshot: 0, batch_size: 16`

|     Task     |Version| Metric | Value  |   |Stderr|
|--------------|------:|--------|-------:|---|-----:|
|arc_easy      |      0|acc     |  0.4024|±  |0.0101|
|              |       |acc_norm|  0.3788|±  |0.0100|
|boolq         |      1|acc     |  0.6199|±  |0.0085|
|lambada_openai|      0|ppl     |111.9939|±  |4.6906|
|              |       |acc     |  0.2354|±  |0.0059|
|openbookqa    |      0|acc     |  0.1440|±  |0.0157|
|              |       |acc_norm|  0.2760|±  |0.0200|
|piqa          |      0|acc     |  0.5713|±  |0.0115|
|              |       |acc_norm|  0.5664|±  |0.0116|
|winogrande    |      0|acc     |  0.5201|±  |0.0140|

|    Task     |Version| Metric |Value |   |Stderr|
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge|      0|acc     |0.1971|±  |0.0116|
|             |       |acc_norm|0.2278|±  |0.0123|

|  Task   |Version| Metric |Value |   |Stderr|
|---------|------:|--------|-----:|---|-----:|
|hellaswag|      0|acc     |0.2618|±  |0.0088|
|         |       |acc_norm|0.2797|±  |0.0090|

|    Task     |Version|Metric|Value |   |Stderr|
|-------------|------:|------|-----:|---|-----:|
|truthfulqa_mc|      1|mc1   |0.2509|±  |0.0152|
|             |       |mc2   |0.4492|±  |0.0156|

---

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.00014
- train_batch_size: 16
- eval_batch_size: 16
- seed: 17514
- gradient_accumulation_steps: 8
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-06
- lr_scheduler_type: inverse_sqrt
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 2.0

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 3.0681        | 0.03  | 150  | 3.0689          | 0.4259   |
| 3.0113        | 0.07  | 300  | 3.0433          | 0.4278   |
| 2.9468        | 0.1   | 450  | 3.0362          | 0.4288   |
| 3.0162        | 0.13  | 600  | 3.0148          | 0.4326   |
| 2.9531        | 0.17  | 750  | 3.0012          | 0.4341   |
| 2.9282        | 0.2   | 900  | 2.9923          | 0.4358   |
| 2.9485        | 0.23  | 1050 | 2.9845          | 0.4357   |
| 2.9365        | 0.27  | 1200 | 2.9749          | 0.4375   |

...

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 2.8215        | 1.7   | 7650 | 2.8943          | 0.4496   |
| 2.7714        | 1.74  | 7800 | 2.8914          | 0.4501   |
| 2.8132        | 1.77  | 7950 | 2.8913          | 0.4500   |
| 2.8505        | 1.8   | 8100 | 2.8906          | 0.4502   |
| 2.8294        | 1.84  | 8250 | 2.8901          | 0.4502   |
| 2.7977        | 1.87  | 8400 | 2.8891          | 0.4499   |
| 2.7501        | 1.9   | 8550 | 2.8878          | 0.4505   |
| 2.8038        | 1.94  | 8700 | 2.8883          | 0.4504   |
| 2.7547        | 1.97  | 8850 | 2.8876          | 0.4502   |

---