Anyone has a code for FLOP calculation of an epoch?

username: Cagri_Kaplan
cooked: <p>need to calculate flops during training (feedforward+backpropagation).<br>
I would appreciate if anyone has the code to do so.</p>

username: smth
cooked: <p>there isn’t an official way to do this, but some folks have written ways of approximating it:</p>
<p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/Lyken17/pytorch-OpCounter" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/7783214?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Lyken17/pytorch-OpCounter" target="_blank" rel="nofollow noopener">Lyken17/pytorch-OpCounter</a></h3>

<p>Count the FLOPs of your PyTorch model. Contribute to Lyken17/pytorch-OpCounter development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/sovrasov/flops-counter.pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/5373517?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/sovrasov/flops-counter.pytorch" target="_blank" rel="nofollow noopener">sovrasov/flops-counter.pytorch</a></h3>

<p>Flops counter for convolutional networks in pytorch framework - sovrasov/flops-counter.pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/1adrianb/pytorch-estimate-flops" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/2567045?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/1adrianb/pytorch-estimate-flops" target="_blank" rel="nofollow noopener">1adrianb/pytorch-estimate-flops</a></h3>

<p>Estimate/count FLOPS for a given neural network using pytorch - 1adrianb/pytorch-estimate-flops</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Cagri_Kaplan
cooked: <p>Thank you, but looks like they only consider the case when model is feed-forwarded or the inference time.<br>
I need something to calculate FLOPs for both backpropagation and feed-forward.<br>
Or am I wrong, these codes also work for backpropagation?</p>

username: AtenaNguyen
cooked: <p>More likely the first one. Most papers and code are considered about complexity of the inference process</p>

username: Mostafa_Elhoushi
cooked: <p>Open AI have a report on training FLOPs of various models:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://openai.com/favicon.png" class="site-icon" width="16" height="16">
      <a href="https://openai.com/blog/ai-and-compute/" target="_blank" title="03:59PM - 16 May 2018" rel="nofollow noopener">OpenAI – 16 May 18</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:128/93;"><img src="https://openai.com/content/images/2019/11/ai-and-compute-all-2.png" class="thumbnail" width="128" height="93"></div>

<h3><a href="https://openai.com/blog/ai-and-compute/" target="_blank" rel="nofollow noopener">AI and Compute</a></h3>

<p>We’re releasing an analysis showing that since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.4-month doubling time (by comparison, Moore’s Law had a 2-year doubling period).</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>and they used the following equation to calculate training FLOPs:</p>
<pre><code class="lang-auto">(add-multiplies per forward pass) * (2 FLOPs/add-multiply) * (3 for forward and backward pass) * (number of examples in dataset) * (number of epochs)
</code></pre>