p-doom/crowd-code-dataset-0.2 huggingface dataset:
[tab] examples/reward_redistribution.html (html)
<!--
  Copyright 2018 The Distill Template Authors

  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!doctype html>

<head>
  <script src="template.v2.js"></script>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta charset="utf8">
  <link rel="icon" type="image/png" href="favicon.png">
</head>

<body>
  <!--
  <distill-header></distill-header>
  -->
  <d-front-matter>
    <script id='distill-front-matter' type="text/json">{
    "title": "PPO On Verification Values All Tokens Equally",
    "description": "When using PPO on verification signals, PPO's value function learns to assign the same value to all tokens in a sequence: the sequence-level probability of successful verification.",
    "published": "February 12, 2025",
    "url": "https://pdoom.org/ppo.html",
    "authors": [
      {
        "author":"Franz Srambical",
        "authorURL":"https://srambical.fr/",
        "affiliations": [{"name": "p(doom)", "url": "https://pdoom.org/"}]
      }
    ],
    "katex": {
      "delimiters": [
        {"left": "$$", "right": "$$", "display": false}
      ]
    }
  }</script>
  </d-front-matter>
  <d-title>
    <p>
        When using PPO on verification signals, PPO's value function learns to assign the same value to all tokens in a sequence: the sequence-level probability of successful verification.
    </p>
  </d-title>
  <d-byline></d-byline>
  <d-article>
    <a class="marker" href="#section-1" id="section-1"><span>1</span></a>
    <p>
      The GAE paper <d-cite key="schulman2015high"></d-cite> defines generalized advantage as
      $$\hat{A}_t^{GAE(\gamma,\lambda)} := \sum_{l=0}^{\infty}(\gamma \lambda)^l \delta^V_{t+l}$$, which, in the LLM
      post-training regime, reduces to $$\hat{A}_t := r_{T} - V(s_t)$$ <d-cite key="srambical2025ppo"></d-cite>.
    </p>

    <p>
      The value function is usually learnt using a regression loss: $$\mathcal{L} = \sum_{t=1}^T ||V_\phi(s_t) - \sum_{l=0}^{\infty} r_{t+l}||^2$$,
      which in our setting reduces to $$\mathcal{L} = \sum_{t=1}^T ||V_\phi(s_t) - r_N||^2$$, hence the learnt value function is .

    </p>
  </d-article>

  <d-appendix>

    <h3>Contributions</h3>
    <p>FS worked on all aspects of this post, including research, analysis and writing.</p>
    <d-bibliography src="bibliography.bib"></d-bibliography>
    <distill-appendix>
    </distill-appendix>
  </d-appendix>

  <distill-footer></distill-footer>

</body>
[content] examples/crowd_code.html (html)

        The most straightforward way to teach a model to do something is to give it a dataset to behaviour-clone off of. No software engineer in the world writes code linearly,
        in a single branch, in a single commit, in a single PR, without jumping around the codebase, without making mistakes, without debugging. crowd-code is our first attempt
        at capturing a broad spectrum of human software engineering, including character-level text insertions, deletions, undo, redo, cursor movement, file switches, jumping
        to function defintions, git checkouts, terminal command execution, autocompletes, LLM changes, and more.
[content] examples/crowd_code.html (html)

        Beyond behaviour-cloning on a dataset crowd-sourced using crowd-code, we eventually want to use crowd-code to annotate the entirety of IDE screencasts on the internet 
        using an inverse dynamics model trained on screen recordings paired with crowd-code's IDE action annotations. This would unlock an entirely new trove of training data
        for software engineering agents.
[content] examples/crowd_code.html (html)

    <p>
        Beyond behaviour-cloning on a dataset crowd-sourced using crowd-code, we eventually want to use crowd-code to annotate the entirety of IDE screencasts on the internet 
        using an inverse dynamics model trained on screen recordings paired with crowd-code's IDE action annotations. This would unlock an entirely new trove of training data
        for software engineering agents.
    </p>
[content] examples/blog.html (html)

          <div class="post-preview">
              <div class="metadata">
                  <div class="publishedDate">April 29, 2025</div>
              </div>
              <a href="ppo_off_policy.html">
                  <div class="description">
                      <h2 class="title">PPO Is An Off-Policy Algorithm</h2>
                      <p class="authors">Franz&nbsp;Srambical, Mihir&nbsp;Mahajan</p>
                      <p class="abstract">PPO is commonly referred to as an on-policy algorithm. We argue that this is confusing, and show that truly on-policy PPO reduces to vanilla policy gradient REINFORCE with baseline.</p>
                  </div>
              </a>
          </div>
[content] examples/crowd_code.html (html)

    <figure style="grid-column: page; margin: 1rem 0; display: flex; justify-content: center"><img src="t5_masking.png"
      style="width:100%; border: 1px solid rgba(0, 0, 0, 0.2);" /></figure>
    <figcaption style="grid-column: page; text-align: center; margin-bottom: 2rem; font-size: 0.8em; color: rgba(0, 0, 0, 0.5);">Figure 1: A schematic of full attention, causal masking, and prefix masking. Figure from <d-cite key="raffel2020exploring"></d-cite>. Used under CC-BY 4.0 license.</figcaption>
[tab] examples/reward_redistribution.html (html)
<!--
  Copyright 2018 p(doom)

  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!doctype html>

<head>
  <script src="template.v2.js"></script>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta charset="utf8">
  <link rel="icon" type="image/png" href="favicon.png">
</head>

<body>
  <!--
  <distill-header></distill-header>
  -->
  <d-front-matter>
    <script id='distill-front-matter' type="text/json">{
    "title": "PPO On Verification Values All Tokens Equally",
    "description": "When using PPO on verification signals, PPO's value function learns to assign the same value to all tokens in a sequence: the sequence-level probability of successful verification.",
    "published": "February 12, 2025",
    "url": "https://pdoom.org/ppo.html",
    "authors": [
      {
        "author":"Franz Srambical",
        "authorURL":"https://srambical.fr/",
        "affiliations": [{"name": "p(doom)", "url": "https://pdoom.org/"}]
      }
    ],
    "katex": {
      "delimiters": [
        {"left": "$$", "right": "$$", "display": false}
      ]
    }
  }</script>
  </d-front-matter>
  <d-title>
    <p>
        When using PPO on verification signals, PPO's value function learns to assign the same value to all tokens in a sequence: the sequence-level probability of successful verification.
    </p>
  </d-title>
  <d-byline></d-byline>
  <d-article>
    <a class="marker" href="#section-1" id="section-1"><span>1</span></a>
    <p>
      The GAE paper <d-cite key="schulman2015high"></d-cite> defines generalized advantage as
      $$\hat{A}_t^{GAE(\gamma,\lambda)} := \sum_{l=0}^{\infty}(\gamma \lambda)^l \delta^V_{t+l}$$, which, in the LLM
      post-training regime, reduces to $$\hat{A}_t := r_{T} - V(s_t)$$ <d-cite key="srambical2025ppo"></d-cite>.
    </p>

    <p>
      The value function is usually learnt using a regression loss: $$\mathcal{L} = \sum_{t=1}^T ||V_\phi(s_t) - \sum_{l=0}^{\infty} r_{t+l}||^2$$,
      which in our setting reduces to $$\mathcal{L} = \sum_{t=1}^T ||V_\phi(s_t) - r_N||^2$$, hence the learnt value function is .

    </p>
  </d-article>

  <d-appendix>

    <h3>Contributions</h3>
    <p>FS worked on all aspects of this post, including research, analysis and writing.</p>
    <d-bibliography src="bibliography.bib"></d-bibliography>
    <distill-appendix>
    </distill-appendix>
  </d-appendix>

  <distill-footer></distill-footer>

</body>
[tab] dist/reward_redistribution.html (html)
<!--
  Copyright 2018 p(doom)

  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!doctype html>

<head>
  <script src="template.v2.js"></script>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta charset="utf8">
  <link rel="icon" type="image/png" href="favicon.png">
</head>

<body>
  <!--
  <distill-header></distill-header>
  -->
  <d-front-matter>
    <script id='distill-front-matter' type="text/json">{
    "title": "PPO On Verification Values All Tokens Equally",
    "description": "When using PPO on verification signals, PPO's value function learns to assign the same value to all tokens in a sequence: the sequence-level probability of successful verification.",
    "published": "February 12, 2025",
    "url": "https://pdoom.org/ppo.html",
    "authors": [
      {
        "author":"Franz Srambical",
        "authorURL":"https://srambical.fr/",
        "affiliations": [{"name": "p(doom)", "url": "https://pdoom.org/"}]
      }
    ],
    "katex": {
      "delimiters": [
        {"left": "$$", "right": "$$", "display": false}
      ]
    }
  }</script>
  </d-front-matter>
  <d-title>
    <p>
        When using PPO on verification signals, PPO's value function learns to assign the same value to all tokens in a sequence: the sequence-level probability of successful verification.
    </p>
  </d-title>
  <d-byline></d-byline>
  <d-article>
    <a class="marker" href="#section-1" id="section-1"><span>1</span></a>
    <p>
      The GAE paper <d-cite key="schulman2015high"></d-cite> defines generalized advantage as
      $$\hat{A}_t^{GAE(\gamma,\lambda)} := \sum_{l=0}^{\infty}(\gamma \lambda)^l \delta^V_{t+l}$$, which, in the LLM
      post-training regime, reduces to $$\hat{A}_t := r_{T} - V(s_t)$$ <d-cite key="srambical2025ppo"></d-cite>.
    </p>

    <p>
      The value function is usually learnt using a regression loss: $$\mathcal{L} = \sum_{t=1}^T ||V_\phi(s_t) - \sum_{l=0}^{\infty} r_{t+l}||^2$$,
      which in our setting reduces to $$\mathcal{L} = \sum_{t=1}^T ||V_\phi(s_t) - r_N||^2$$, hence the learnt value function is .

    </p>
  </d-article>

  <d-appendix>

    <h3>Contributions</h3>
    <p>FS worked on all aspects of this post, including research, analysis and writing.</p>
    <d-bibliography src="bibliography.bib"></d-bibliography>
    <distill-appendix>
    </distill-appendix>
  </d-appendix>

  <distill-footer></distill-footer>

</body>