Text Generation
Transformers
PyTorch
Safetensors
code
Eval Results
Inference Endpoints

updated metadata (results from table 2)

#2
by rootacess - opened
Files changed (1) hide show
  1. README.md +190 -0
README.md CHANGED
@@ -37,6 +37,196 @@ model-index:
37
  type: pass@1
38
  value: 39.2
39
  verified: false
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  ---
41
 
42
  ![Octopack](https://github.com/bigcode-project/octopack/blob/31f3320f098703c7910e43492c39366eeea68d83/banner.png?raw=true)
 
37
  type: pass@1
38
  value: 39.2
39
  verified: false
40
+ - task:
41
+ type: text-generation
42
+ dataset:
43
+ type: bigcode/humanevalpack
44
+ name: HumanEvalSynthesize Java
45
+ metrics:
46
+ - name: pass@1
47
+ type: pass@1
48
+ value: 38.2
49
+ verified: false
50
+ - task:
51
+ type: text-generation
52
+ dataset:
53
+ type: bigcode/humanevalpack
54
+ name: HumanEvalSynthesize Go
55
+ metrics:
56
+ - name: pass@1
57
+ type: pass@1
58
+ value: 30.4
59
+ verified: false
60
+ - task:
61
+ type: text-generation
62
+ dataset:
63
+ type: bigcode/humanevalpack
64
+ name: HumanEvalSynthesize C++
65
+ metrics:
66
+ - name: pass@1
67
+ type: pass@1
68
+ value: 35.6
69
+ verified: false
70
+ - task:
71
+ type: text-generation
72
+ dataset:
73
+ type: bigcode/humanevalpack
74
+ name: HumanEvalSynthesize Rust
75
+ metrics:
76
+ - name: pass@1
77
+ type: pass@1
78
+ value: 23.4
79
+ verified: false
80
+ - task:
81
+ type: text-generation
82
+ dataset:
83
+ type: bigcode/humanevalpack
84
+ name: HumanEvalSynthesize Avg.
85
+ metrics:
86
+ - name: pass@1
87
+ type: pass@1
88
+ value: 35.5
89
+ verified: false
90
+ - task:
91
+ type: text-generation
92
+ dataset:
93
+ type: bigcode/humanevalpack
94
+ name: HumanEvalExplain Python
95
+ metrics:
96
+ - name: pass@1
97
+ type: pass@1
98
+ value: 35.1
99
+ verified: false
100
+ - task:
101
+ type: text-generation
102
+ dataset:
103
+ type: bigcode/humanevalpack
104
+ name: HumanEvalExplain JavaScript
105
+ metrics:
106
+ - name: pass@1
107
+ type: pass@1
108
+ value: 24.5
109
+ verified: false
110
+ - task:
111
+ type: text-generation
112
+ dataset:
113
+ type: bigcode/humanevalpack
114
+ name: HumanEvalExplain Java
115
+ metrics:
116
+ - name: pass@1
117
+ type: pass@1
118
+ value: 27.3
119
+ verified: false
120
+ - task:
121
+ type: text-generation
122
+ dataset:
123
+ type: bigcode/humanevalpack
124
+ name: HumanEvalExplain Go
125
+ metrics:
126
+ - name: pass@1
127
+ type: pass@1
128
+ value: 21.1
129
+ verified: false
130
+ - task:
131
+ type: text-generation
132
+ dataset:
133
+ type: bigcode/humanevalpack
134
+ name: HumanEvalExplain C++
135
+ metrics:
136
+ - name: pass@1
137
+ type: pass@1
138
+ value: 24.1
139
+ verified: false
140
+ - task:
141
+ type: text-generation
142
+ dataset:
143
+ type: bigcode/humanevalpack
144
+ name: HumanEvalExplain Rust
145
+ metrics:
146
+ - name: pass@1
147
+ type: pass@1
148
+ value: 14.8
149
+ verified: false
150
+ - task:
151
+ type: text-generation
152
+ dataset:
153
+ type: bigcode/humanevalpack
154
+ name: HumanEvalExplain Avg.
155
+ metrics:
156
+ - name: pass@1
157
+ type: pass@1
158
+ value: 24.5
159
+ verified: false
160
+ - task:
161
+ type: text-generation
162
+ dataset:
163
+ type: bigcode/humanevalpack
164
+ name: HumanEvalFix Python
165
+ metrics:
166
+ - name: pass@1
167
+ type: pass@1
168
+ value: 30.2
169
+ verified: false
170
+ - task:
171
+ type: text-generation
172
+ dataset:
173
+ type: bigcode/humanevalpack
174
+ name: HumanEvalFix JavaScript
175
+ metrics:
176
+ - name: pass@1
177
+ type: pass@1
178
+ value: 28.4
179
+ verified: false
180
+ - task:
181
+ type: text-generation
182
+ dataset:
183
+ type: bigcode/humanevalpack
184
+ name: HumanEvalFix Java
185
+ metrics:
186
+ - name: pass@1
187
+ type: pass@1
188
+ value: 30.6
189
+ verified: false
190
+ - task:
191
+ type: text-generation
192
+ dataset:
193
+ type: bigcode/humanevalpack
194
+ name: HumanEvalFix Go
195
+ metrics:
196
+ - name: pass@1
197
+ type: pass@1
198
+ value: 30.2
199
+ verified: false
200
+ - task:
201
+ type: text-generation
202
+ dataset:
203
+ type: bigcode/humanevalpack
204
+ name: HumanEvalFix C++
205
+ metrics:
206
+ - name: pass@1
207
+ type: pass@1
208
+ value: 26.1
209
+ verified: false
210
+ - task:
211
+ type: text-generation
212
+ dataset:
213
+ type: bigcode/humanevalpack
214
+ name: HumanEvalFix Rust
215
+ metrics:
216
+ - name: pass@1
217
+ type: pass@1
218
+ value: 16.5
219
+ verified: false
220
+ - task:
221
+ type: text-generation
222
+ dataset:
223
+ type: bigcode/humanevalpack
224
+ name: HumanEvalFix Avg.
225
+ metrics:
226
+ - name: pass@1
227
+ type: pass@1
228
+ value: 27.0
229
+ verified: false
230
  ---
231
 
232
  ![Octopack](https://github.com/bigcode-project/octopack/blob/31f3320f098703c7910e43492c39366eeea68d83/banner.png?raw=true)