yuxiang630 commited on
Commit
cee4ec0
1 Parent(s): ee5afd4

Add evaluation and improve wording

Browse files
README.md CHANGED
@@ -104,7 +104,7 @@ We introduce StarCoder2-15B-Instruct-v0.1, the very first entirely self-aligned
104
 
105
  ### Intended use
106
 
107
- The model is exclusively trained with 50k **singe-turn** instruction-response pairs focused on **code generation**. It is supposed to answer **implementation-related instructions**. Instructions in other formats may result in unexpected outcomes. In such cases, we recommend providing a **response prefix** or a **one-shot example** to guide the model.
108
 
109
  Here is an example to get started with the model using the [transformers](https://huggingface.co/docs/transformers/index) library:
110
 
@@ -169,9 +169,15 @@ def sum_list_of_integers(numbers):
169
 
170
  ### Bias, Risks, and Limitations
171
 
172
- StarCoder2-15B-Instruct-v0.1 is primarily finetuned for Python code generation tasks that can be verified through execution, which may lead to biases and limitations. For example, the model may not accurately follow instructions that constrains the output format and may have limitations in its performance with other programming languages and out-of-domain coding problems. In such cases, we recommend providing a response prefix or a one-shot example to guide the model.
173
 
174
- StarCoder2-15B-Instruct-v0.1 also inherits the bias, risks, and limitations from its base StarCoder2-15B model. For more information, please refer to the [StarCoder2-15B model card](https://huggingface.co/bigcode/starcoder2-15b).
 
 
 
 
 
 
175
 
176
  ## Training Details
177
 
 
104
 
105
  ### Intended use
106
 
107
+ The model is designed to respond to **coding-related instructions** in a single turn. Instructions in other styles may result in less accurate responses
108
 
109
  Here is an example to get started with the model using the [transformers](https://huggingface.co/docs/transformers/index) library:
110
 
 
169
 
170
  ### Bias, Risks, and Limitations
171
 
172
+ StarCoder2-15B-Instruct-v0.1 is primarily finetuned for Python code generation tasks that can be verified through execution, which may lead to certain biases and limitations. For example, the model might not adhere strictly to instructions that dictate the output format. In these situations, it's beneficial to provide a **response prefix** or a **one-shot example** to steer the model’s output. Additionally, the model may have limitations with other programming languages and out-of-domain coding tasks.
173
 
174
+ The model also inherits the bias, risks, and limitations from its base StarCoder2-15B model. For more information, please refer to the [StarCoder2-15B model card](https://huggingface.co/bigcode/starcoder2-15b).
175
+
176
+ ## Evaluation on EvalPlus, LiveCodeBench, and DS-1000
177
+
178
+ ![EvalPlus](assets/sc2-instruct-evalplus.png)
179
+
180
+ ![LiveCodeBench and DS-1000](assets/sc2-instruct-lcb-ds.png)
181
 
182
  ## Training Details
183
 
assets/sc2-instruct-evalplus.png ADDED
assets/sc2-instruct-lcb-ds.png ADDED