hails commited on
Commit
4f3e5b7
1 Parent(s): ce788ba

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -1
README.md CHANGED
@@ -22,7 +22,9 @@ license: apache-2.0
22
 
23
  FIM-1.3B is the first of a series of large-scale infilling-enabled autoregressive language models trained by CarperAI. FIM-1.3B is the first of these models, and future models (both larger and smaller) trained on greater quantities of code data will be released, potentially with different architectural variations optimized for code.
24
 
25
- This is a preliminary release of an experimental artifact and should be treated as such.
 
 
26
 
27
 
28
 
@@ -125,8 +127,10 @@ All 3 models here are trained using the same configuration with differing FIM hy
125
 
126
 
127
 
 
128
 
129
 
 
130
 
131
 
132
 
 
22
 
23
  FIM-1.3B is the first of a series of large-scale infilling-enabled autoregressive language models trained by CarperAI. FIM-1.3B is the first of these models, and future models (both larger and smaller) trained on greater quantities of code data will be released, potentially with different architectural variations optimized for code.
24
 
25
+ This is a preliminary release of an experimental artifact and should be treated as such. We are releasing these results and this model in the hopes that it may be useful to the greater research community, especially those interested in LMs for code and pair programming tools.
26
+
27
+ CarperAI will be releasing larger LMs better tuned for code in the near future, building on these experiments.
28
 
29
 
30
 
 
127
 
128
 
129
 
130
+ Here HumanEval-infilling is reported as Pass@10 with a temperature of 0.8 (such that 100 \* the score reported here = Pass@10 as a percentage), Lambada is reported as perplexity, and all other benchmarks report accuracy as a number between 0 and 1.
131
 
132
 
133
+ These results are subject to change, but appear to indicate that AliBi with FIM does not enable infilling, while rotary positional embeddings do allow for infilling to be learned.
134
 
135
 
136