Add pipeline tag and library name (#1)

- Add pipeline tag and library name (40ec329c4e8a4bb0026cb7bc6260891a929ebaeb)


Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>

README.md

@@ -1,6 +1,9 @@
 ---
 license: mit
+pipeline_tag: reinforcement-learning
+library_name: transformers
 ---
+
 <div align="center">
 
 # Open Reasoner Zero
@@ -24,7 +27,7 @@ An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
     src="https://img.shields.io/badge/Notion-%23000000.svg?style=for-the-badge&logo=notion&logoColor=white"/></a>
 
   <br>
-  <a href="https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/ORZ_paper.pdf"><b>Paper PDF Link [WIP]</b>👁️</a>
+  <a href="https://arxiv.org/abs/2503.24290"><b>Paper Arxiv Link </b>👁️</a>
 </div>
 
 <div>
@@ -34,10 +37,11 @@ An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
 
 ## Overview 🌊
 We introduce **Open-Reasoner-Zero**, the first open source implementation of large-scale reasoning-oriented RL training focusing on scalability, simplicity and accessibility.
+Using the same base model as DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance on AIME2024, MATH500, and the GPQA Diamond benchmark while demonstrating remarkable efficiency—requiring only a tenth of the training steps, compared to DeepSeek-R1-Zero pipeline.
 
 To enable broader participation in this pivotal moment we witnessed and accelerate research towards artificial general intelligence (AGI), 
 we release our source code, parameter settings, training data, and model weights.
-Please refer to our [paper](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/ORZ_paper.pdf) for more insights across various model sizes. 
+Please refer to our [paper](https://arxiv.org/abs/2503.24290) for more insights across various model sizes. 
 
 **Let the Reasoner-Zero tide rise!**
 
@@ -46,7 +50,7 @@ Please refer to our [paper](https://github.com/Open-Reasoner-Zero/Open-Reasoner-
 
 ![](figure/teaser.png)
 
-*Figure 1 | Evaluation performance of Open-Reasoner-Zero-\{7B, 32B\}. Evaluation performance of Open-Reasoner-Zero-\{7B, 32B\} on benchmarks (averaged on 16 responses) during training. Using the same base model as DeepSeek-R1-Zero-Qwen-32B, Open-Reasoner-Zero-32B achieves superior performance on AIME2024, MATH500, and GPQA Diamond benchmark-requiring only a tenth of the training steps.*
+*Figure 1 | Evaluation performance of Open-Reasoner-Zero-{7B, 32B}. Evaluation performance of Open-Reasoner-Zero-{7B, 32B} on benchmarks (averaged on 16 responses) during training. Using the same base model as DeepSeek-R1-Zero-Qwen-32B, Open-Reasoner-Zero-32B achieves superior performance on AIME2024, MATH500, and GPQA Diamond benchmark-requiring only a tenth of the training steps.*
 
 ![](figure/train_curve.png)
 *Figure 2 | Train-time Scale up on Train Reward and Response Length of Open-Reasoner-Zero (ORZ) - \{0.5B, 1.5B, 7B, 32B\}. Train Reward and Response Length increase steadily, demonstrating consistent scalability across model sizes. Interestingly, the ORZ-32B Response Length exhibits fluctuations without negatively impacting training stability, highlighting the robustness of our minimalist recipe.*
@@ -56,7 +60,7 @@ Please refer to our [paper](https://github.com/Open-Reasoner-Zero/Open-Reasoner-
 <strong>[2025/03/31]</strong>
 We announce a major milestone for `Open-Reasoner-Zero`:
 
-- 🌊 [Updated Paper](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/ORZ_paper.pdf) with new results.
+- 🌊 [Updated Paper](https://arxiv.org/abs/2503.24290) with new results.
 - 🔭 [Easy-to-use Training Scripts](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/tree/main/playground):
   - [ORZ-1.5B training scripts](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/playground/orz_1p5b_ppo.py) and [ORZ-0.5B training scripts](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/playground/orz_0p5b_ppo.py) (main results in Figure 2). 
   - [Minimal resource training scripts](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/playground/orz_0p5b_ppo_1gpu.py): ORZ-0.5B can be run on a single A800/H800 gpu!
@@ -75,7 +79,7 @@ We announce a major milestone for `Open-Reasoner-Zero`:
 We release `Open-Reasoner-Zero`. 
 
 As part of this release, we open-source:
-- 🌊 [Paper](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/ORZ_paper.pdf) on our comprehensive analysis and insights in Reasoner-Zero training
+- 🌊 [Paper](https://arxiv.org/abs/2503.24290) on our comprehensive analysis and insights in Reasoner-Zero training
 - 🤗 HF Model [`Open-Reasoner-Zero-7B`](https://huggingface.co/Open-Reasoner-Zero/Open-Reasoner-Zero-7B) and [`Open-Reasoner-Zero-32B`](https://huggingface.co/Open-Reasoner-Zero/Open-Reasoner-Zero-32B)
 - 🎁 [`Our curated 57k training data`](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/tree/main/data)
 - 📄 [Training Scripts](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/tree/main/playground) to enjoy your own Reasoner-Zero journey!
@@ -94,7 +98,7 @@ We release all of curated high-quality training data in the [`data`](https://git
   * [extended 72k](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/data/orz_math_72k_collection_extended.json), mainly cleaned from OpenR1-Math-220k.
 * [hard 13k](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/data/orz_math_13k_collection_hard.json), mined from the first stage of ORZ-32B training.
 
-The details for how to collect data are described in our [paper](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/ORZ_paper.pdf).
+The details for how to collect data are described in our [paper](https://arxiv.org/abs/2503.24290).
 
 ### Installation & Training Scripts
 We release our [Dockerfile](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/docker/Dockerfile) in [docker](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/tree/main/docker) folder to facilitate the reproducibility of our training.
@@ -186,6 +190,14 @@ DEBUG_MODE=True python -m playground.orz_14m_ppo_mini
 DEBUG_MODE=True python -m playground.orz_7b_ppo
 ```
 
+### How to Use the Model
+#### Policy Model
+Policy models can be used in the same way as any chat model in transformers and vllm, since we have put the chat template jinja in the tokenizer.
+
+#### Critic Model
+Critic models can be loaded the same way like in the [training code](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/orz/ppo/actors.py#L738). 
+
+
 ## Acknowledgements 💖 
 
 - This work was supported by computing resources and valuable feedback provided by [StepFun](https://www.stepfun.com/) and Tsinghua University.
@@ -209,11 +221,13 @@ We have several wechat groups to help discussions and sharing, you can scan the
 ## Citation
 
 ```bibtex
-@misc{OpenReasonerZero2025,
-  title={Open-Reasoner-Zero: An Open Source Approach to Scaling Reinforcement Learning on the Base Model},
-  author={Jingcheng Hu and Yinmin Zhang and Qi Han and Daxin Jiang and Xiangyu Zhang, Heung-Yeung Shum},
-  year={2025},
-  howpublished={\url{https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero}},
+@misc{hu2025openreasonerzeroopensourceapproach,
+      title={Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model}, 
+      author={Jingcheng Hu and Yinmin Zhang and Qi Han and Daxin Jiang and Xiangyu Zhang and Heung-Yeung Shum},
+      year={2025},
+      eprint={2503.24290},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2503.24290}, 
 }
-```
-
+```