Correct way to calculate FLOPS in model

kl_divergence: <p>I want to calculate FLOPS of my model for every epoch. I’ve come across few posts and github issues that discuss this but I’m not sure if they are calculating it correctly. I need some suggestions ? If this helps, my model is a transformer.</p>

kl_divergence: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could you please recommend something?</p>

ptrblck: <p>Unfortunately I cannot recommend a particular lib or method, but what are your current concerns about the code you’ve found so far?</p>

kl_divergence: <p>They approximate and provide better support for (Conv+Linear). I don’t have any conv layers, and I want the approach to be reliable, as to not report inconsistent results due to implementation. The <a href="https://github.com/pytorch/pytorch/issues/5013" rel="nofollow noopener">feature request</a> to provide official support from 2018 is still open.</p>

kl_divergence: <p>I looked in more carefully, their implementation does not account for several modules like <code>nn.Embedding</code> and <code>LayerNorm</code>, <code>GeLU()</code> etc. These approaches may not give an accurate count of FLOPS. I’m referring to the ones Soumith recommended <a href="https://discuss.pytorch.org/t/anyone-has-a-code-for-flop-calculation-of-an-epoch/49666/2">here</a></p>

Yaroslav_Bulatov: <p>FLOP count is a property of an algorithm rather than a model. Does Linear layer have 2mqp or mq(2p-1) FLOPs? Depends how matmul is performed – see discussion <a href="https://github.com/tensorflow/tensorflow/pull/19792" rel="nofollow noopener">here</a>. You can get an approximate count by assuming some reference implementation.</p>
<p><code>nn.Embedding</code> is a dictionary lookup, so technically it has 0 FLOPS.</p>
<p>Since FLOP count is going to be approximate anyway, you only care about the heaviest to compute layers. You could profile your model and see if there are any expensive layers not covered already. TensorFlow has some reference formulas <a href="https://github.com/tensorflow/tensorflow/search?q=RegisterStatistics&amp;unscoped_q=RegisterStatistics" rel="nofollow noopener">here</a></p>

kl_divergence: <p>Thanks for your response. Also could you please clarify the relation between MACs and FLOPS?</p>

YaraAlnaggar: <p>I guess this answers your question<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/sovrasov/flops-counter.pytorch/issues/16" target="_blank" rel="nofollow noopener">github.com/sovrasov/flops-counter.pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/sovrasov/flops-counter.pytorch/issues/16" target="_blank" rel="nofollow noopener">What is the relationship between GMACs and GFLOPs?</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-08-06" data-time="07:33:14" data-timezone="UTC">07:33AM - 06 Aug 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/nizhenliang" target="_blank" rel="nofollow noopener">
          <img alt="nizhenliang" src="https://avatars2.githubusercontent.com/u/48742170?v=4" class="onebox-avatar-inline" width="20" height="20">
          nizhenliang
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Why use the GMACs？GMACs is different with GFLOPs.</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">question</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


sudonto: <p>Hi <a class="mention" href="/u/kl_divergence">@kl_divergence</a>,  I am thinking the same with y ou and I found it: <a href="https://github.com/facebookresearch/SlowFast/blob/0cc82440fee6e51a5807853b583be238bf26a253/slowfast/utils/misc.py#L106" rel="nofollow noopener">https://github.com/facebookresearch/SlowFast/blob/0cc82440fee6e51a5807853b583be238bf26a253/slowfast/utils/misc.py#L106</a></p>
<p>It is official account of fbresearch so I guess we can rely on it (?). Have a look.</p>

ppwwyyxx: <p>Our team at Facebook AI computer vision has released a tool to compute and summarize the flop count of any pytorch model: <a href="https://github.com/facebookresearch/fvcore/blob/master/docs/flop_count.md" class="inline-onebox" rel="noopener nofollow ugc">fvcore/flop_count.md at master · facebookresearch/fvcore · GitHub</a>. Please check it out!</p>

sio277: <p>Thanks for the nice work.<br>
FLOPS count for the spectral normalization is also supported?<br>
(<a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.spectral_norm.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.utils.spectral_norm — PyTorch 1.8.1 documentation</a>)</p>

ppwwyyxx: <p>This normalization is by default ignored but it should take negligible flops anyway.</p>

Santhoshnumberone: <aside class="quote no-group" data-username="ppwwyyxx" data-post="10" data-topic="67198">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ppwwyyxx/48/3031_2.png" class="avatar"> ppwwyyxx:</div>
<blockquote>
<p>fvcore/flop_count.md at master · facebookresearch/fvcore · GitHub</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ppwwyyxx">@ppwwyyxx</a> How do I find the FLOPS of a pre trained pytorch object detection model?</p>
<pre><code class="lang-auto">from fvcore.nn import FlopCountAnalysis
import torchvision
from PIL import Image
import torch
import torchvision.transforms as T

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False)
img_path = "/content/drive/MyDrive/dataset/images/fr-1751.jpg"
img = Image.open(img_path)
transform = T.Compose([T.ToTensor()])
img = transform(img)
# pred = model([img])
flops = FlopCountAnalysis(model, [img])
flops.total()
</code></pre>
<p>I get this error</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-56-a72097dab463&gt; in &lt;module&gt;()
     12 # pred = model([img])
     13 flops = FlopCountAnalysis(model, [img])
---&gt; 14 flops.total()
     15 
     16 flops.by_operator()

9 frames
/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py in total(self, module_name)
    246             int : The aggregated statistic.
    247         """
--&gt; 248         stats = self._analyze()
    249         module_name = self.canonical_module_name(module_name)
    250         total_count = sum(stats.counts[module_name].values())

/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py in _analyze(self)
    549             elif self._warn_trace == "no_tracer_warning":
    550                 warnings.filterwarnings("ignore", category=TracerWarning)
--&gt; 551             graph = _get_scoped_trace_graph(self._model, self._inputs, self._aliases)
    552 
    553         # Assures even modules not in the trace graph are initialized to zero count

/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py in _get_scoped_trace_graph(module, inputs, aliases)
    174         register_hooks(mod, name)
    175 
--&gt; 176     graph, _ = _get_trace_graph(module, inputs)
    177 
    178     for handle in hook_handles:

/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py in _get_trace_graph(f, args, kwargs, strict, _force_outplace, return_inputs, _return_inputs_states)
   1164     if not isinstance(args, tuple):
   1165         args = (args,)
-&gt; 1166     outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
   1167     return outs

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py in forward(self, *args)
    130             _create_interpreter_name_lookup_fn(),
    131             self.strict,
--&gt; 132             self._force_outplace,
    133         )
    134 

/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py in wrapper(*args)
    116             if self._return_inputs_states:
    117                 inputs_states.append(_unflatten(in_args, in_desc))
--&gt; 118             outs.append(self.inner(*trace_inputs))
    119             if self._return_inputs_states:
    120                 inputs_states[0] = (inputs_states[0], trace_inputs)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1126             input = bw_hook.setup_input_hook(input)
   1127 
-&gt; 1128         result = forward_call(*input, **kwargs)
   1129         if _global_forward_hooks or self._forward_hooks:
   1130             for hook in (*_global_forward_hooks.values(), *self._forward_hooks.values()):

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _slow_forward(self, *input, **kwargs)
   1096                 recording_scopes = False
   1097         try:
-&gt; 1098             result = self.forward(*input, **kwargs)
   1099         finally:
   1100             if recording_scopes:

/usr/local/lib/python3.7/dist-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
     59         """
     60         if self.training and targets is None:
---&gt; 61             raise ValueError("In training mode, targets should be passed")
     62         if self.training:
     63             assert targets is not None

ValueError: In training mode, targets should be passed
</code></pre>

ppwwyyxx: <p>The error said it very well:</p>
<blockquote>
<p>ValueError: In training mode, targets should be passed</p>
</blockquote>
<p>Set <code>model.eval()</code> in order to make predictions with the model.</p>

Santhoshnumberone: <aside class="quote no-group" data-username="ppwwyyxx" data-post="10" data-topic="67198">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ppwwyyxx/48/3031_2.png" class="avatar"> ppwwyyxx:</div>
<blockquote>
<p>fvcore/flop_count.md at master · facebookresearch/fvcore · GitHub</p>
</blockquote>
</aside>
<p>It did work. Thank you</p>

H_Jahan: <p>could you please let me know what is the formula behind the flops calculation of batch normalization layer?</p>

harisgulzar1: <p>I have tried multiple tools including fvcore, and many of them didn’t work for several submodules of my model.<br>
Then I found <strong>deepspeed</strong> and it works seamlessly with a very detailed analysis of each submodule.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.deepspeed.ai/tutorials/flops-profiler/">
  <header class="source">

      <a href="https://www.deepspeed.ai/tutorials/flops-profiler/" target="_blank" rel="noopener nofollow ugc" title="02:22PM - 01 March 2023">DeepSpeed – 1 Mar 23</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://www.deepspeed.ai/tutorials/flops-profiler/" target="_blank" rel="noopener nofollow ugc">Flops Profiler</a></h3>

  <p>Measure the parameters, latency, and floating-point operations of your model</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Really appreciate their work</p>