Resetting cache in benchmark

AidanGoldfarb: <p>I am attempting to benchmark some things with torch.compile, including the overhead of compilation in different modes. I am calling <code>dynamo.reset()</code> before each call to torch.compile, however this does not seem to be clearing the cache.</p>
<p>Below is the code I am executing, <code>mul</code> is a simple function, <code>timeit</code> simply passes arguments to its argument and calls the function, timing it . Whichever mode is called first has overhead of ~1 sec, the following three have orders of magnitude less.</p>
<p>Please let me know if I am clearing the cache incorrectly, thank you.</p>
<pre><code class="lang-auto">dynamo.reset()
def_cmptm, mul_def = timeit( torch.compile, args=mul, kwargs={"mode": "default",} )

dynamo.reset()
max_cmptm, mul_max = timeit( torch.compile, args=mul, kwargs={"mode": "max-autotune"} )

dynamo.reset()
red_cmptm, mul_red = timeit( torch.compile, args=mul, kwargs={"mode": "reduce-overhead"} )

dynamo.reset()
max_nc_cmptm, mul_max_nc = timeit( torch.compile, args=mul, kwargs={"mode": "max-autotune-no-cudagraphs"} )
</code></pre>

AidanGoldfarb: <p>It appears the first run is simply the compiler warming up. Inserting a dummy compilation solves this. Interestingly, <code>dynamo.reset()</code> still has no effect on compile times. This surprises me, as I would think some information about a previously compiled function would be cached.</p>

bhack: <p>But it seems that the team is using <code>dynamo.reset()</code> to clear the cache:</p><aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/FBGEMM/pull/1992">
  <header class="source">

      <a href="https://github.com/pytorch/FBGEMM/pull/1992" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/FBGEMM</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/pytorch/FBGEMM/pull/1992" target="_blank" rel="noopener nofollow ugc">Clear torch.compile cache between tests</a>
      </h4>

    <div class="branches">
      <code>pytorch:main</code> ← <code>anijain2305:export-D48940544</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2023-09-04" data-time="02:21:59" data-timezone="UTC">02:21AM - 04 Sep 23 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/anijain2305" target="_blank" rel="noopener nofollow ugc">
            <img alt="anijain2305" src="https://avatars.githubusercontent.com/u/13822661?v=4" class="onebox-avatar-inline" width="20" height="20">
            anijain2305
          </a>
        </div>

        <div class="lines" title="1 commits changed 1 files with 21 additions and 0 deletions">
          <a href="https://github.com/pytorch/FBGEMM/pull/1992/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+21</span>
            <span class="removed">-0</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Summary: Preparing for reducing cache size work.

Differential Revision: D489405<span class="show-more-container"><a href="https://github.com/pytorch/FBGEMM/pull/1992" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">44</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


AidanGoldfarb: <p>Perhaps; perhaps it will be documented someday :-). I figured it may have to do with the recompilation cache.</p>

nd21: <p>I am facing a similar issue. My model’s peak memory usage for the first run is higher than the second run. And second run onwards the peak memory is somewhat the same. I am wondering what may be the reason for this?</p>