File size: 1,779 Bytes
c2d7af9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c486ffc
c2d7af9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a4acee6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1e7b711
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
---
license: llama3.1
language:
- en
base_model:
- nvidia/OpenMath2-Llama3.1-8B
pipeline_tag: text-generation
tags:
- math
- nvidia
- llama
---

## GGUF quantized version of OpenMath2-Llama3.1-8B

project original [source](https://huggingface.co/nvidia/OpenMath2-Llama3.1-8B) (finetuned model)

Q_2_K (not nice)

Q_3_K_S (acceptable)

Q_3_K_M is acceptable (good for running with CPU)

Q_3_K_L (acceptable)

Q_4_K_S (okay)

Q_4_K_M is recommanded (balance)

Q_5_K_S (good)

Q_5_K_M (good in general)

Q_6_K is good also; if you want a better result; take this one instead of Q_5_K_M

Q_8_0 which is very good; need a reasonable size of RAM otherwise you might expect a long wait

f16 is similar to the original hf model; opt this one or hf also fine; make sure you have a good machine

### how to run it

use any connector for interacting with gguf; i.e., [gguf-connector](https://pypi.org/project/gguf-connector/)

  <style>
      .image-container {
          display: flex;
          justify-content: center;
          align-items: center;
          gap: 20px;
      }
      .image-container img {
          width: 350px;
          height: auto;
      }
  </style>

<div class="image-container">
    <img src="https://huggingface.co/nvidia/OpenMath2-Llama3.1-8B/resolve/main/scaling_plot.jpg" title="Performance of Llama-3.1-8B-Instruct as it is trained on increasing proportions of OpenMathInstruct-2">
    <img src="https://huggingface.co/nvidia/OpenMath2-Llama3.1-8B/resolve/main/math_level_comp.jpg" title="Comparison of OpenMath2-Llama3.1-8B vs. Llama-3.1-8B-Instruct across MATH levels">
</div>

the chart and figure above are from finetuned model (nvidia side); those are used for comparing between the finetuned model and the base model; and the base model is from meta