p-doom/crowd-code-dataset-0.2 huggingface dataset:
[tab] main.tex (latex)
\documentclass[11pt, a4paper, logo, onecolumn, copyright, nonumbering]{deepmind}

\usepackage[authoryear, sort&compress, round]{natbib}
\bibliographystyle{abbrvnat}
\input{math_commands}

\usepackage{floatrow}
\usepackage{varwidth}
\usepackage[]{algorithm2e}
\usepackage{algpseudocode}
\usepackage{bbm}
\usepackage{wrapfig}
\usepackage{siunitx}

	itle{Reinforced Self-Training (
est) for Language Modeling}%

\correspondingauthor{ca9lar@gmail.com}

\keywords{Offline RL, reinforcement learning, RL from human feedback, language, natural language processing, machine translation}


ewcommand{
est}{\emph{ReST}}

ewcommand{\grow}{	exttt{Grow}}

ewcommand{\improve}{	exttt{Improve}}

ewcommand{\BC}{	exttt{BC}}


ewcommand{\caglar}[1]{{\color{red} (Caglar : #1)}}

ewcommand{
ando}[1]{{\color{red} (Nando : #1)}}

ewcommand{\srivatsan}[1]{{\color{green} (Srivatsan : #1)}}

ewcommand{	om}[1]{{\color{blue} (Tom : #1)}}

ewcommand{\ksenia}[1]{{\color{cyan} Ksenia : #1}}

\definecolor{dm-blue-500}{RGB}{0, 69, 177}
\definecolor{dm-purple-500}{RGB}{105,50,230}


eportnumber{001} %

\author{
{\Authfont
Caglar Gulcehre	extsuperscript{*	extdagger,1},
Tom Le Paine	extsuperscript{*	extdagger,1},
Srivatsan Srinivasan	extsuperscript{*	extdagger,1},
Ksenia Konyushkova	extsuperscript{	extdagger,1}, 
Lotte Weerts	extsuperscript{	extdagger,1}}
\leavevmode \protect\\[1em]
{\Authfont
Abhishek Sharma	extsuperscript{	extdagger,1}, 
Aditya Siddhant	extsuperscript{	extdagger,1}, 
Alex Ahern	extsuperscript{1}, 
Miaosen Wang	extsuperscript{1},
Chenjie Gu	extsuperscript{1}}, 
\leavevmode \protect\\[1em]
{\Authfont
Wolfgang Macherey	extsuperscript{2},
Arnaud Doucet	extsuperscript{1},
Orhan Firat	extsuperscript{	extdagger,1}, 
Nando de Freitas	extsuperscript{1}}
\leavevmode \protect\\[1em]
{\Affilfont
	extsuperscript{*}Contributed equally, 	extsuperscript{	extdagger}Core contributors\protect\\
	extsuperscript{1}Google DeepMind, 	extsuperscript{2}Google Research 
}}

\input{0-abstract}

\begin{document}
\maketitle
\input{1-intro}
\input{2-rest}
\input{4-results}
\input{3-relwork}
\input{5-discussion}

\paragraph{Acknowledgements}

We would like to thank the members from the machine translation teams at Google and Google DeepMind for their inputs to the project during the brainstorming phases and for setting up the codebase that this project was developed upon \citep{yu2020wmt}. We would like to thank Matt Hoffman, Bobak Shahriari, Taylan Cemgil and Chris Dyer for the discussions about this project. We are grateful for the feedback provided by Bilal Piot for an early draft of this paper. We would also like to thank those responsible for various different frameworks that we used during the project such as the DeepMind JAX ecosystem \citep{deepmind2020jax} and Launchpad \citep{yang2021launchpad}. 

\bibliography{main}
\input{6-appendix}
\end{document}
[tab] 0-abstract.tex (latex)
\begin{abstract}

Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call 	extit{Reinforced Self-Training} (
est{}). Given an initial LLM policy, 
est{} produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. 
est{} is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While 
est{} is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that 
est{} can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

\end{abstract}
[content] main.tex (latex)
We would like to thank the members from the machine translation teams at Google and Google DeepMind for their inputs to the project during the brainstorming phases and for setting up the codebase that this project was developed upon \citep{yu2020wmt}. We would like to thank Matt Hoffman, Bobak Shahriari, Taylan Cemgil and Chris Dyer for the discussions about this project. We are grateful for the feedback provided by Bilal Piot for an early draft of this paper. We would also like to thank those responsible for various different frameworks that we used during the project such as the DeepMind JAX ecosystem \citep{deepmind2020jax} and Launchpad \citep{yang2021launchpad}.
[content] main.tex (latex)
Abhishek Sharma	extsuperscript{	extdagger,1}, 
Aditya Siddhant	extsuperscript{	extdagger,1}, 
Alex Ahern	extsuperscript{1}, 
Miaosen Wang	extsuperscript{1},
Chenjie Gu	extsuperscript{1}}, 
\leavevmode \protect\\[1em]
{\Authfont
[tab] /Users/franzsrambical/Downloads/arXiv-2308.08998v2/0-abstract.tex (latex)
\begin{abstract}

Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call 	extit{Reinforced Self-Training} (
est{}). Given an initial LLM policy, 
est{} produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. 
est{} is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While 
est{} is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that 
est{} can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

\end{abstract}
[content] main.tex (latex)
\begin{abstract}
%World models are increasingly positioned as a pathway to overcoming data scarcity in domains such as robotics, yet their feasibility as training environments for agents remains uncertain. A key unanswered question is how compute and data requirements scale with environment complexity. We introduce Jasmine, a scalable, high-performance JAX-based codebase designed as the foundation for systematic investigation of empirical environment complexity scaling trends. Jasmine supports training world models with full reproducibility, scales from single hosts to hundreds of accelerators with minimal code changes, and includes performance optimizations across dataloading, training and checkpointing. By pairing Jasmine with curated large-scale datasets, we establish the infrastructure for rigorous benchmarking pipelines across model families and architectural ablations.
While world models are increasingly positioned as a pathway to overcoming data scarcity in domains such as robotics, open training infrastructure for world modeling remains nascent. We introduce Jasmine, a performant JAX-based world modeling codebase that scales from single hosts to hundreds of accelerators with minimal code changes. Jasmine achieves an order-of-magnitude faster reproduction of the CoinRun case study compared to prior open implementations, enabled by performance optimizations across data loading, training and checkpointing. The codebase guarantees fully reproducible training and supports diverse sharding configurations. By pairing Jasmine with curated large-scale datasets, we establish infrastructure for rigorous benchmarking pipelines across model families and architectural ablations.
\end{abstract}
[content] main.tex (latex)

\section{Related Work}
Although research on world models with its inception decades ago \citep{sutton1991dyna} has matured over the years \citep{ha2018world,hafner2019learning,hafner2019dream,hafner2020mastering,hafner2023mastering,alonso2024diffusion}, they have only been scaled up recently \citep{hafner2025training,bruce2024genie,parkerholder2024genie2,deepmind2025genie3,valevski2024diffusion,agarwal2025cosmos,hu2023gaia,guo2025mineworld,decart2024oasis,lucid2024lucidv1,li2025hunyuan,pearce2024scaling}. While the open training ecosystem in language modeling provides mature solutions for large-scale language pretraining \citep{megatron-lm,maxtext-library}, open training infrastructure for world modeling is still nascent \citep{wayfarer_labs_owl_wms,Savov_2025_CVPR}. Closest to our work is \citet{willi2024jafar}, an open-source reproduction of Genie \citep{bruce2024genie}, which we build upon and significantly extend.
[tab] main.out (plaintext)
\BOOKMARK [1][-]{section.1}{\376\377\000I\000n\000t\000r\000o\000d\000u\000c\000t\000i\000o\000n}{}% 1
\BOOKMARK [1][-]{section.2}{\376\377\000J\000a\000s\000m\000i\000n\000e}{}% 2
\BOOKMARK [1][-]{section.3}{\376\377\000E\000x\000p\000e\000r\000i\000m\000e\000n\000t\000s}{}% 3
\BOOKMARK [1][-]{section.4}{\376\377\000R\000e\000l\000a\000t\000e\000d\000\040\000W\000o\000r\000k\000\040\000a\000n\000d\000\040\000D\000i\000s\000c\000u\000s\000s\000i\000o\000n}{}% 4
\BOOKMARK [1][-]{appendix.A}{\376\377\000C\000o\000i\000n\000r\000u\000n\000\040\000C\000a\000s\000e\000\040\000S\000t\000u\000d\000y}{}% 5
\BOOKMARK [1][-]{appendix.B}{\376\377\000A\000b\000l\000a\000t\000i\000o\000n\000s}{}% 6
\BOOKMARK [1][-]{appendix.C}{\376\377\000D\000i\000f\000f\000u\000s\000i\000o\000n\000\040\000B\000a\000s\000e\000l\000i\000n\000e}{}% 7
\BOOKMARK [1][-]{appendix.D}{\376\377\000B\000i\000t\000w\000i\000s\000e\000\040\000D\000e\000t\000e\000r\000m\000i\000n\000i\000s\000m}{}% 8
\BOOKMARK [1][-]{appendix.E}{\376\377\000H\000y\000p\000e\000r\000p\000a\000r\000a\000m\000e\000t\000e\000r\000\040\000C\000o\000n\000f\000i\000g\000u\000r\000a\000t\000i\000o\000n\000s}{}% 9
\BOOKMARK [1][-]{appendix.F}{\376\377\000E\000x\000t\000e\000n\000d\000i\000n\000g\000\040\000M\000a\000s\000k\000G\000I\000T\000\040\000t\000o\000\040\000V\000i\000d\000e\000o\000s}{}% 10
\BOOKMARK [1][-]{appendix.G}{\376\377\000E\000v\000a\000l\000u\000a\000t\000i\000o\000n\000\040\000M\000e\000t\000r\000i\000c\000s}{}% 11
\BOOKMARK [1][-]{appendix.H}{\376\377\000J\000a\000f\000a\000r\000'\000s\000\040\000B\000a\000t\000c\000h\000e\000d\000\040\000M\000a\000s\000k\000i\000n\000g\000\040\000L\000o\000g\000i\000c}{}% 12
\BOOKMARK [1][-]{appendix.I}{\376\377\000M\000o\000d\000e\000l\000\040\000I\000n\000s\000p\000e\000c\000t\000i\000o\000n\000\040\000u\000s\000i\000n\000g\000\040\000T\000r\000e\000e\000s\000c\000o\000p\000e}{}% 13
[content] main.tex (latex)
Related Work}
Although research on world models with its inception decades ago \citep{sutton1991dyna} has matured over the years \citep{ha2018world,hafner2019learning,hafner2019dream,hafner2020mastering,hafner2023mastering,alonso2024diffusion}, they have only been scaled up recently \citep{hafner2025training,bruce2024genie,parkerholder2024genie2,deepmind2025genie3,valevski2024diffusion,agarwal2025cosmos,hu2023gaia,guo2025mineworld,decart2024oasis,lucid2024lucidv1,li2025hunyuan,pearce2024scaling}. While the open training ecosystem in language modeling provides mature solutions for large-scale language pretraining \citep{megatron-lm,maxtext-library}, open training infrastructure for world modeling is still nascent \citep{wayfarer_labs_owl_wms,Savov_2025_CVPR}. Closest to our work is \citet{willi2024jafar}, an open-source reproduction of Genie \citep{bruce2024genie}, which we build upon and significantly extend.

\section{
[content] main.tex (latex)
\section{Introduction}

Over the past decades, the field of deep learning has increasingly been shaped by methods that leverage vast data troves \citep{jozefowicz2016exploring, radford2018improving, radford2019language, chowdhery2022palm, raffel2020exploring, JMLR:v23:21-0998, deng2009imagenet}, and paradigms that unlock new ones \citep{guo2025deepseek, christiano2017deep, radford2021learning, srambical2025crowd-sourcing, silver2016mastering, berner2019dota}. Internet-scale pre-training, preference modeling, and reinforcement learning using verification signals offer a compelling pathway for language models to attain human-level performance \citep{LuongLockhart2025GeminiIMO, LinCheng2025GeminiICPC}, yet data is increasingly bottlenecking progress from spiky towards general intelligence. While some do
[content] main.tex (latex)
%World models are increasingly positioned as a pathway to overcoming data scarcity in domains such as robotics, yet their feasibility as training environments for agents remains uncertain. A key unanswered question is how compute and data requirements scale with environment complexity. We introduce Jasmine, a scalable, high-performance JAX-based codebase designed as the foundation for systematic investigation of empirical environment complexity scaling trends. Jasmine supports training world models with full reproducibility, scales from single hosts to hundreds of accelerators with minimal code changes, and includes performance optimizations across dataloading, training and checkpointing. By pairing Jasmine with curated large-scale datasets, we establish the infrastructure for rigorous benchmarking pipelines across model families and architectural ablations.
[content] main.tex (latex)

ewcommand{
est}{\emph{ReST}}

ewcommand{\grow}{	exttt{Grow}}

ewcommand{\improve}{	exttt{Improve}}

ewcommand{\BC}{	exttt{BC}}


ewcommand{\caglar}[1]{{\color{red} (Caglar : #1)}}

ewcommand{
ando}[1]{{\color{red} (Nando : #1)}}

ewcommand{\srivatsan}[1]{{\color{green} (Srivatsan : #1)}}

ewcommand{	om}[1]{{\color{blue} (Tom : #1)}}

ewcommand{\ksenia}[1]{{\color{cyan} Ksenia : #1}}

\definecolor{dm-blue-500}{RGB}{0, 69, 177}
\definecolor{dm-purple-500}{RGB}{105,50,230}
[content] main.tex (latex)
\begin{abstract}
    %World models are increasingly positioned as a pathway to overcoming data scarcity in domains such as robotics, yet their feasibility as training environments for agents remains uncertain. A key unanswered question is how compute and data requirements scale with environment complexity. We introduce Jasmine, a scalable, high-performance JAX-based codebase designed as the foundation for systematic investigation of empirical environment complexity scaling trends. Jasmine supports training world models with full reproducibility, scales from single hosts to hundreds of accelerators with minimal code changes, and includes performance optimizations across dataloading, training and checkpointing. By pairing Jasmine with curated large-scale datasets, we establish the infrastructure for rigorous benchmarking pipelines across model families and architectural ablations.
    While world models are increasingly positioned as a pathway to overcoming data scarcity in domains such as robotics, open training infrastructure for world modeling remains nascent. We introduce Jasmine, a performant JAX-based world modeling codebase that scales from single hosts to hundreds of accelerators with minimal code changes. Jasmine achieves an order-of-magnitude faster reproduction of the CoinRun case study compared to prior open implementations, enabled by performance optimizations across data loading, training and checkpointing. The codebase guarantees fully reproducible training and supports diverse sharding configurations. By pairing Jasmine with curated large-scale datasets, we establish infrastructure for rigorous benchmarking pipelines across model families and architectural ablations.
    \end{abstract}
[tab] /Users/franzsrambical/Downloads/arXiv-2308.08998v2/main.tex (latex)
\documentclass[11pt, a4paper, logo, onecolumn, copyright, nonumbering]{deepmind}

\usepackage[authoryear, sort&compress, round]{natbib}
\bibliographystyle{abbrvnat}
\input{math_commands}

\usepackage{floatrow}
\usepackage{varwidth}
\usepackage[]{algorithm2e}
\usepackage{algpseudocode}
\usepackage{bbm}
\usepackage{wrapfig}
\usepackage{siunitx}

	itle{Reinforced Self-Training (
est) for Language Modeling}%

\correspondingauthor{ca9lar@gmail.com}

\keywords{Offline RL, reinforcement learning, RL from human feedback, language, natural language processing, machine translation}


ewcommand{
est}{\emph{ReST}}

ewcommand{\grow}{	exttt{Grow}}

ewcommand{\improve}{	exttt{Improve}}

ewcommand{\BC}{	exttt{BC}}


ewcommand{\caglar}[1]{{\color{red} (Caglar : #1)}}

ewcommand{
ando}[1]{{\color{red} (Nando : #1)}}

ewcommand{\srivatsan}[1]{{\color{green} (Srivatsan : #1)}}

ewcommand{	om}[1]{{\color{blue} (Tom : #1)}}

ewcommand{\ksenia}[1]{{\color{cyan} Ksenia : #1}}

\definecolor{dm-blue-500}{RGB}{0, 69, 177}
\definecolor{dm-purple-500}{RGB}{105,50,230}


eportnumber{001} %

\author{
{\Authfont
Caglar Gulcehre	extsuperscript{*	extdagger,1},
Tom Le Paine	extsuperscript{*	extdagger,1},
Srivatsan Srinivasan	extsuperscript{*	extdagger,1},
Ksenia Konyushkova	extsuperscript{	extdagger,1}, 
Lotte Weerts	extsuperscript{	extdagger,1}}
\leavevmode \protect\\[1em]
{\Authfont
Abhishek Sharma	extsuperscript{	extdagger,1}, 
Aditya Siddhant	extsuperscript{	extdagger,1}, 
Alex Ahern	extsuperscript{1}, 
Miaosen Wang	extsuperscript{1},
Chenjie Gu	extsuperscript{1}}, 
\leavevmode \protect\\[1em]
{\Authfont
Wolfgang Macherey	extsuperscript{2},
Arnaud Doucet	extsuperscript{1},
Orhan Firat	extsuperscript{	extdagger,1}, 
Nando de Freitas	extsuperscript{1}}
\leavevmode \protect\\[1em]
{\Affilfont
	extsuperscript{*}Contributed equally, 	extsuperscript{	extdagger}Core contributors\protect\\
	extsuperscript{1}Google DeepMind, 	extsuperscript{2}Google Research 
}}

\input{0-abstract}

\begin{document}
\maketitle
\input{1-intro}
\input{2-rest}
\input{4-results}
\input{3-relwork}
\input{5-discussion}

\paragraph{Acknowledgements}

We would like to thank the members from the machine translation teams at Google and Google DeepMind for their inputs to the project during the brainstorming phases and for setting up the codebase that this project was developed upon \citep{yu2020wmt}. We would like to thank Matt Hoffman, Bobak Shahriari, Taylan Cemgil and Chris Dyer for the discussions about this project. We are grateful for the feedback provided by Bilal Piot for an early draft of this paper. We would also like to thank those responsible for various different frameworks that we used during the project such as the DeepMind JAX ecosystem \citep{deepmind2020jax} and Launchpad \citep{yang2021launchpad}. 

\bibliography{main}
\input{6-appendix}
\end{document}