Add question-answering pipeline tag and github repo link

README.md

@@ -1,16 +1,18 @@
 ---
-license: apache-2.0
+base_model:
+- Qwen/Qwen2.5-Math-7B-Instruct
 language:
 - en
+library_name: transformers
+license: apache-2.0
 metrics:
 - accuracy
-base_model:
-- Qwen/Qwen2.5-Math-7B-Instruct
-library_name: transformers
+pipeline_tag: question-answering
 ---
+
 ## SuperCorrect: Supervising and Correcting Language Models with Error-Driven Insights
 
-> [SuperCorrect: Supervising and Correcting Language Models with Error-Driven Insights](link)
+> [SuperCorrect: Supervising and Correcting Language Models with Error-Driven Insights](https://arxiv.org/abs/2410.09008)
 > [Ling Yang\*](https://yangling0818.github.io/), [Zhaochen Yu*](https://github.com/BitCodingWalkin), [Tianjun Zhang](https://tianjunz.github.io/), [Minkai Xu](https://minkaixu.com/), [Joseph E. Gonzalez](https://people.eecs.berkeley.edu/~jegonzal/),[Bin Cui](https://cuibinpku.github.io/), [Shuicheng Yan](https://yanshuicheng.info/)
 >
 > Peking University, Skywork AI, UC Berkeley, Stanford University 
@@ -103,25 +105,131 @@ response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
 print(response)
 ```
 
-## Performance
+#### 🔥 vLLM
+
+```python
+import os
+from vllm import LLM, SamplingParams
+model_name = 'BitStarWalkin/SuperCorrect-7B'
+hierarchical_prompt = "Solve the following math problem in a step-by-step XML format, each step should be enclosed within tags like <Step1></Step1>. For each step enclosed within the tags, determine if this step is challenging and tricky, if so, add detailed explanation and analysis enclosed within <Key> </Key> in this step, as helpful annotations to help you thinking and remind yourself how to conduct reasoning correctly. After all the reasoning steps, summarize the common solution and reasoning steps to help you and your classmates who are not good at math generalize to similar problems within <Generalized></Generalized>. Finally present the final answer within <Answer> </Answer>."
+prompts = [
+    "For what positive value of $t$ is $|{-4+ti}| = 6$?",
+    "Find the distance between the foci of the ellipse \\[9x^2 + \\frac{y^2}{9} = 99.\\]",
+    "The fourth term of a geometric series is $24$ and the eleventh term is $3072$. What is the common ratio?"
+]
+combined_prompts = [hierarchical_prompt + '\n' + prompt for prompt in prompts]
+sampling_params = SamplingParams(temperature=0, top_p=1,max_tokens=1024)
+llm = LLM(model=model_name, trust_remote_code=True)
+outputs = llm.generate(combined_prompts, sampling_params)
+
+#Print the outputs.
+for output in outputs:
+    prompt = output.prompt
+    generated_text = output.outputs[0].text
+    print(f"Prompt: {prompt}")
+    print(f"Generated text: {generated_text}")
+```
+
+Here we also provide inference code with [vLLM](https://github.com/vllm-project/vllm) . vLLM is a fast and easy-to-use library for LLM inference and serving.
+
 
-We evaluate our SupperCorrect-7B on two widely used English math benchmarks GSM8K and MATH. All evaluations are tested with our evaluation method which is zero-shot hierarchical thought based prompting.
+### 1. Our evaluation  
 
-![image](table.png)
+Here we provide two different evaluation methods: **online version**  which utilizes GPT-4o to conduct a more fair and robust judgement and **offline version**  which utilizes programming method to verify the final results. Both methods aim to provide a more accurate and strict evaluation results, as the final results in MATH dataset are not always numeric or pure expression. We now provide online version for evaluation, we will update soon for offline version.
 
+
+```bash
+API_KEY= "Input your key here"
+MODEL_NAME_OR_PATH="BitStarWalkin/SuperCorrect-7B"
+export CUDA_VISIBLE_DEVICES="0"
+bash evaluation.sh $API_KEY $MODEL_NAME_OR_PATH
+
+```
+
+
+
+### 2. Evaluation with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
+
+```bash
+lm_eval --model hf \
+    --model_args pretrained="Qwen2.5-Math-7B-Instruct" \
+    --tasks minerva_math \
+    --log_samples \
+    --output_path Qwen2.5-Math-7B-Instruct-lm-evaluation \
+    --batch_size 12
+
+lm_eval --model hf \
+    --model_args pretrained="SuperCorrect-7B" \
+    --tasks minerva_math \
+    --log_samples \
+    --output_path SuperCorrect-7B-lm-evaluation \
+    --batch_size 12
+```
+Evaluation results produced by lm-evaluation:
+
+| Qwen2.5-Math-7B-Instruct    | Version | Filter | n-shot | Metric      |      |  Value |      | Stderr |
+| ----------------------------------- | ------: | ------ | -----: | ----------- | ---- | -----: | ---- | -----: |
+| minerva_math                        |       1 | none   |      4 | exact_match | ↑    | 0.5034 | ±    | 0.0064 |
+| - minerva_math_algebra              |       1 | none   |      4 | exact_match | ↑    | 0.7009 | ±    | 0.0133 |
+| - minerva_math_counting_and_prob    |       1 | none   |      4 | exact_match | ↑    | 0.5232 | ±    | 0.0230 |
+| - minerva_math_geometry             |       1 | none   |      4 | exact_match | ↑    | 0.4635 | ±    | 0.0228 |
+| - minerva_math_intermediate_algebra |       1 | none   |      4 | exact_match | ↑    | 0.2237 | ±    | 0.0139 |
+| - minerva_math_num_theory           |       1 | none   |      4 | exact_match | ↑    | 0.4667 | ±    | 0.0215 |
+| - minerva_math_prealgebra           |       1 | none   |      4 | exact_match | ↑    | 0.7394 | ±    | 0.0149 |
+| - minerva_math_precalc              |       1 | none   |      4 | exact_match | ↑    | 0.2143 | ±    | 0.0176 |
+
+
+
+| SuperCorrect-7B              | Version | Filter | n-shot | Metric      |      |  Value |      | Stderr |
+| ------------------------------------ | ------: | ------ | -----: | ----------- | ---- | -----: | ---- | -----: |
+| minerva_math                         |       1 | none   |      4 | exact_match | ↑    | 0.6188 (**+0.1154**) | ±    | 0.0065 |
+| - minerva_math_algebra               |       1 | none   |      4 | exact_match | ↑    | 0.7936 (**+0.0927**) | ±    | 0.0118 |
+| - minerva_math_counting_and_prob     |       1 | none   |      4 | exact_match | ↑    | 0.5802 (**+0.0570**) | ±    | 0.0227 |
+| - minerva_math_geometry              |       1 | none   |      4 | exact_match | ↑    | 0.5261 (**+0.0626**) | ±    | 0.0228 |
+| - minerva_math_intermediate_algebra  |       1 | none   |      4 | exact_match | ↑    | 0.4385 (**+0.2148**) | ±    | 0.0165 |
+| - minerva_math_num_theory            |       1 | none   |      4 | exact_match | ↑    | 0.6167 (**+0.1500**) | ±    | 0.0209 |
+| - minerva_math_prealgebra            |       1 | none   |      4 | exact_match | ↑    | 0.7715 (**+0.0321**) | ±    | 0.0142 |
+| - minerva_math_precalc               |       1 | none   |      4 | exact_match | ↑    | 0.4103 (**+0.1960**) | ±    | 0.0211 |
+
+
+| Summary       | Version | Filter | n-shot | Metric      |      |  Value |      | Stderr |
+| ------------ | ------: | ------ | ------ | ----------- | ---- | -----: | ---- | -----: |
+| Qwen2.5-Math-7B-Instruct |       1 | none   |       4| exact_match | ↑    | 0.5034 | ±    | 0.0064 |
+| SuperCorrect-7B  |       1 | none   |       4| exact_match | ↑    | 0.6188 (**+0.1154**) | ±    | 0.0065 |
+
+### 3. Evaluation with [Qwen2.5-Math-Evaluation](https://github.com/QwenLM/Qwen2.5-Math)
+```bash
+export CUDA_VISIBLE_DEVICES="0"
+MODEL_NAME_OR_PATH="Qwen/Qwen2.5-Math-7B-Instruct"
+bash sh/eval.sh $PROMPT_TYPE $MODEL_NAME_OR_PATH
+
+export CUDA_VISIBLE_DEVICES="0"
+MODEL_NAME_OR_PATH="BitStarWalkin/SuperCorrect-7B"
+bash sh/eval.sh $PROMPT_TYPE $MODEL_NAME_OR_PATH
+```
+Evaluation results produced by Qwen2.5-Math-Eval:
+| Model            | MATH Accuracy (%) |
+| ---------------- | ----------------- |
+| Qwen2.5-Math     | 80.6              |
+| **SuperCorrect**     | **82.1**              |
+| **Our Improvement**   | **+1.5**          |
+
+## Code
+This model and evaluation results are based on the code at https://github.com/YangLing0818/SuperCorrect-llm
 ## Citation
 
 ```bash
-@article{yang2024supercorrect,
-title={SuperCorrect: Supervising and Correcting Language Models with Error-Driven Insights}
+@article{yang2025supercorrect,
+  title={SuperCorrect: Supervising and Correcting Language Models with Error-Driven Insights},
   author={Yang, Ling and Yu, Zhaochen and Zhang, Tianjun and Xu, Minkai and Gonzalez, Joseph E and Cui, Bin and Yan, Shuicheng},
-  journal={arXiv preprint arXiv:2410.09008},
-  year={2024}
+  booktitle={International Conference on Learning Representations},
+  year={2025}
 }
+
 @article{yang2024buffer,
   title={Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models},
   author={Yang, Ling and Yu, Zhaochen and Zhang, Tianjun and Cao, Shiyi and Xu, Minkai and Zhang, Wentao and Gonzalez, Joseph E and Cui, Bin},
-  journal={arXiv preprint arXiv:2406.04271},
+  journal={Advances in Neural Information Processing Systems},
   year={2024}
 }
 ```