Triangle104 commited on
Commit
4156e91
1 Parent(s): 82dc07c

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +130 -0
README.md ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: ibm-granite/granite-8b-code-instruct-4k
3
+ datasets:
4
+ - bigcode/commitpackft
5
+ - TIGER-Lab/MathInstruct
6
+ - meta-math/MetaMathQA
7
+ - glaiveai/glaive-code-assistant-v3
8
+ - glaive-function-calling-v2
9
+ - bugdaryan/sql-create-context-instruction
10
+ - garage-bAInd/Open-Platypus
11
+ - nvidia/HelpSteer
12
+ library_name: transformers
13
+ license: apache-2.0
14
+ metrics:
15
+ - code_eval
16
+ pipeline_tag: text-generation
17
+ tags:
18
+ - code
19
+ - granite
20
+ - llama-cpp
21
+ - gguf-my-repo
22
+ inference: false
23
+ model-index:
24
+ - name: granite-8b-code-instruct-4k
25
+ results:
26
+ - task:
27
+ type: text-generation
28
+ dataset:
29
+ name: HumanEvalSynthesis(Python)
30
+ type: bigcode/humanevalpack
31
+ metrics:
32
+ - type: pass@1
33
+ value: 57.9
34
+ name: pass@1
35
+ - type: pass@1
36
+ value: 52.4
37
+ name: pass@1
38
+ - type: pass@1
39
+ value: 58.5
40
+ name: pass@1
41
+ - type: pass@1
42
+ value: 43.3
43
+ name: pass@1
44
+ - type: pass@1
45
+ value: 48.2
46
+ name: pass@1
47
+ - type: pass@1
48
+ value: 37.2
49
+ name: pass@1
50
+ - type: pass@1
51
+ value: 53.0
52
+ name: pass@1
53
+ - type: pass@1
54
+ value: 42.7
55
+ name: pass@1
56
+ - type: pass@1
57
+ value: 52.4
58
+ name: pass@1
59
+ - type: pass@1
60
+ value: 36.6
61
+ name: pass@1
62
+ - type: pass@1
63
+ value: 43.9
64
+ name: pass@1
65
+ - type: pass@1
66
+ value: 16.5
67
+ name: pass@1
68
+ - type: pass@1
69
+ value: 39.6
70
+ name: pass@1
71
+ - type: pass@1
72
+ value: 40.9
73
+ name: pass@1
74
+ - type: pass@1
75
+ value: 48.2
76
+ name: pass@1
77
+ - type: pass@1
78
+ value: 41.5
79
+ name: pass@1
80
+ - type: pass@1
81
+ value: 39.0
82
+ name: pass@1
83
+ - type: pass@1
84
+ value: 32.9
85
+ name: pass@1
86
+ ---
87
+
88
+ # Triangle104/granite-8b-code-instruct-4k-Q4_K_M-GGUF
89
+ This model was converted to GGUF format from [`ibm-granite/granite-8b-code-instruct-4k`](https://huggingface.co/ibm-granite/granite-8b-code-instruct-4k) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
90
+ Refer to the [original model card](https://huggingface.co/ibm-granite/granite-8b-code-instruct-4k) for more details on the model.
91
+
92
+ ## Use with llama.cpp
93
+ Install llama.cpp through brew (works on Mac and Linux)
94
+
95
+ ```bash
96
+ brew install llama.cpp
97
+
98
+ ```
99
+ Invoke the llama.cpp server or the CLI.
100
+
101
+ ### CLI:
102
+ ```bash
103
+ llama-cli --hf-repo Triangle104/granite-8b-code-instruct-4k-Q4_K_M-GGUF --hf-file granite-8b-code-instruct-4k-q4_k_m.gguf -p "The meaning to life and the universe is"
104
+ ```
105
+
106
+ ### Server:
107
+ ```bash
108
+ llama-server --hf-repo Triangle104/granite-8b-code-instruct-4k-Q4_K_M-GGUF --hf-file granite-8b-code-instruct-4k-q4_k_m.gguf -c 2048
109
+ ```
110
+
111
+ Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.
112
+
113
+ Step 1: Clone llama.cpp from GitHub.
114
+ ```
115
+ git clone https://github.com/ggerganov/llama.cpp
116
+ ```
117
+
118
+ Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
119
+ ```
120
+ cd llama.cpp && LLAMA_CURL=1 make
121
+ ```
122
+
123
+ Step 3: Run inference through the main binary.
124
+ ```
125
+ ./llama-cli --hf-repo Triangle104/granite-8b-code-instruct-4k-Q4_K_M-GGUF --hf-file granite-8b-code-instruct-4k-q4_k_m.gguf -p "The meaning to life and the universe is"
126
+ ```
127
+ or
128
+ ```
129
+ ./llama-server --hf-repo Triangle104/granite-8b-code-instruct-4k-Q4_K_M-GGUF --hf-file granite-8b-code-instruct-4k-q4_k_m.gguf -c 2048
130
+ ```