Calculate Flops in Pytorch and Tensorflow are not equal?

hcleung3325: <p>Given the same model, I found that the calculated flops in pytorch and tensorflow are different. I used the keras_flops (<a href="https://pypi.org/project/keras-flops/" class="inline-onebox" rel="noopener nofollow ugc">keras-flops · PyPI</a>) in tensorflow, and ptflops (<a href="https://pypi.org/project/ptflops/" class="inline-onebox" rel="noopener nofollow ugc">ptflops · PyPI</a>) in pytorch to calculate flops. Is that TensorFlow has some tricks to speed up the computation so that few flops are measured? How come pytorch and tensorflow can have different flops with the same model?</p>
<p>The FLops in pytorch is</p>
<pre><code class="lang-auto">Model_1(
  0.013 M, 100.000% Params, 45.486 GMac, 100.000% MACs, 
  (begin): Sequential(
    0.002 M, 11.804% Params, 0.851 GMac, 1.870% MACs, 
    (0): Conv2d(0.001 M, 11.367% Params, 0.819 GMac, 1.801% MACs, 1, 56, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): PReLU(0.0 M, 0.437% Params, 0.032 GMac, 0.069% MACs, num_parameters=56)
  )
  (middle): Sequential(
    0.007 M, 52.775% Params, 3.803 GMac, 8.360% MACs, 
    (0): Conv2d(0.001 M, 5.340% Params, 0.385 GMac, 0.846% MACs, 56, 12, kernel_size=(1, 1), stride=(1, 1))
    (1): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (2): Conv2d(0.001 M, 10.212% Params, 0.736 GMac, 1.618% MACs, 12, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (4): Conv2d(0.001 M, 10.212% Params, 0.736 GMac, 1.618% MACs, 12, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (5): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (6): Conv2d(0.001 M, 10.212% Params, 0.736 GMac, 1.618% MACs, 12, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (8): Conv2d(0.001 M, 10.212% Params, 0.736 GMac, 1.618% MACs, 12, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): PReLU(0.0 M, 0.094% Params, 0.007 GMac, 0.015% MACs, num_parameters=12)
    (10): Conv2d(0.001 M, 5.684% Params, 0.409 GMac, 0.900% MACs, 12, 56, kernel_size=(1, 1), stride=(1, 1))
    (11): PReLU(0.0 M, 0.437% Params, 0.032 GMac, 0.069% MACs, num_parameters=56)
  )
  (final): ConvTranspose2d(0.005 M, 35.420% Params, 40.833 GMac, 89.770% MACs, 56, 1, kernel_size=(9, 9), stride=(4, 4), padding=(4, 4), output_padding=(3, 3))
)
Computational complexity:       45.49 GMac

</code></pre>
<p>My model in tensorflow</p>
<pre><code class="lang-auto">d=56
s=12

inp = Input((750 ,750, 1))
x = Conv2D(d, (5,5), padding='same')(inp)
x = PReLU()(x)

x = Conv2D(s, (1,1), padding='valid')(x)
x = PReLU()(x)

x = Conv2D(s, (3,3), padding='same')(x)
x = PReLU()(x)
x = Conv2D(s, (3,3), padding='same')(x)
x = PReLU()(x)
x = Conv2D(s, (3,3), padding='same')(x)
x = PReLU()(x)
x = Conv2D(s, (3,3), padding='same')(x)
x = PReLU()(x)

x = Conv2D(d, (1,1), padding='same')(x)
x = PReLU()(x)
out = Conv2DTranspose(1 ,(9,9), strides=(4, 4),padding='same',output_padding = 3)(x)
</code></pre>
<p>The Flops output in tensorflow is: Profile:</p>
<pre><code class="lang-auto">node name | # float_ops
Conv2D                   8.92b float_ops (100.00%, 61.95%)
Conv2DBackpropInput      5.10b float_ops (38.05%, 35.44%)
Neg                      180.00m float_ops (2.61%, 1.25%)
BiasAdd                  105.75m float_ops (1.36%, 0.73%)
Mul                      90.00m float_ops (0.63%, 0.63%)

======================End of Report==========================
The FLOPs is:14.3 GFlops
</code></pre>

eqy: <p>It looks like the conv2d transpose is responsible for the bulk of the flops in the PyTorch result. Can you verify that this number makes sense?</p>

hcleung3325: <p>Thanks.<br>
The flops for deconvolution is:<br>
Cout * (1+Cin * k * k) * Hout * Wout<br>
= 1 * (1+56 * 9 * 9) * 3000 * 3000<br>
= 40.83 GFlops.<br>
(<a href="https://titanwolf.org/Network/Articles/Article?AID=abffa95e-ee67-428f-99f0-f3c0af9adeee#gsc.tab=0" class="inline-onebox" rel="noopener nofollow ugc">Calculating a convolution neural network computation float(Others-Community)</a>)<br>
I think pytorch calculate right, but tensorflow didn’t.<br>
Just wonder why?</p>

hcleung3325: <p>Can anyone help plz?</p>

ptrblck: <aside class="quote no-group" data-username="hcleung3325" data-post="3" data-topic="121810">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hcleung3325/48/38499_2.png" class="avatar"> hcleung3325:</div>
<blockquote>
<p>I think pytorch calculate right, but tensorflow didn’t.<br>
Just wonder why?</p>
</blockquote>
</aside>
<p>Bugs can happen, and since you’ve checked the calculation and think the TF calculation is wrong, I’m sure the TF devs would be happy about an issue (and a fix). <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

hcleung3325: <p>Thank you for reply.<br>
Do you think the pytorch flops calculation and my calculation are correct?<br>
It seems to be weird that only tensorflow has problem on calculating flops.</p>

hcleung3325: <p>Also, is there any pytorch tools can calculate the flops for Pixelshuffle operation?<br>
I cannot find a equation to calculate it myself.<br>
Thanks.</p>

Mohammad_Shahpouri: <p>Hello <a class="mention" href="/u/hcleung3325">@hcleung3325</a> I do not know if it is still a problem. However, tensorflow computes FLOPs, while tools that compute FLOPs for pytorch actually calculate MACs.</p>
<p><strong>Disclaimer:</strong>  I can not claim to be familiar with all pytorch tools. However, I am aware that <a href="https://github.com/zhijian-liu/torchprofile" rel="noopener nofollow ugc">torchprofile</a> and <a href="https://github.com/facebookresearch/fvcore/tree/main" rel="noopener nofollow ugc">fvcore</a> are capable of computing MACs.</p>
<h1><a name="pytorch-1" class="anchor" href="#pytorch-1"></a><strong>Pytorch</strong></h1>
<p>According to <a href="https://medium.com/@pashashaik/a-guide-to-hand-calculating-flops-and-macs-fa5221ce5ccc" rel="noopener nofollow ugc">medium post</a>, it is possible to compute MACs and FLOPs using the mentioned method. Therefore, I believe you calculated MACs that were close to pytorch. I would like to suggest using <a href="https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md" rel="noopener nofollow ugc">fvcore</a>, the official/semi-official flops counter provided by <a href="https://github.com/facebookresearch/fvcore/tree/main" rel="noopener nofollow ugc">facebook/meta</a>, instead of <a href="https://github.com/zhijian-liu/torchprofile" rel="noopener nofollow ugc">torchprofile</a> which was employed in <a href="https://medium.com/@pashashaik/a-guide-to-hand-calculating-flops-and-macs-fa5221ce5ccc" rel="noopener nofollow ugc">medium post</a>.</p>
<pre><code class="lang-auto">!pip -q install fvcore
</code></pre>
<pre><code class="lang-auto">import torch
import torchvision
from fvcore.nn import FlopCountAnalysis

convnexttiny_weights = torchvision.models.ConvNeXt_Tiny_Weights.IMAGENET1K_V1
model = torchvision.models.convnext_tiny(weights=convnexttiny_weights)

inputs = (torch.randn(1, 3, 224, 224), )

macs = FlopCountAnalysis(model, inputs)
macs = macs.total()
flops = macs * 2
print(f'MACs = {macs / 1e+9:,} G')
print(f'FLOPs = {flops / 1e+9:,} G')
</code></pre>
<pre><code class="lang-auto">MACs = 4.470437376 G
FLOPs = 8.940874752 G
</code></pre>
<h2><a name="your-model-2" class="anchor" href="#your-model-2"></a><strong>Your model</strong></h2>
<pre><code class="lang-auto">d=56
s=12
model = torch.nn.Sequential(
            torch.nn.Conv2d(in_channels=1, out_channels=d, kernel_size=(5, 5), padding=2),
            torch.nn.PReLU(),

            torch.nn.Conv2d(in_channels=d, out_channels=s, kernel_size=(1, 1), padding=0),
            torch.nn.PReLU(),

            torch.nn.Conv2d(in_channels=s, out_channels=s, kernel_size=(3 ,3), padding=1),
            torch.nn.PReLU(),

            torch.nn.Conv2d(in_channels=s, out_channels=s, kernel_size=(3 ,3), padding=1),
            torch.nn.PReLU(),

            torch.nn.Conv2d(in_channels=s, out_channels=s, kernel_size=(3 ,3), padding=1),
            torch.nn.PReLU(),

            torch.nn.Conv2d(in_channels=s, out_channels=s, kernel_size=(3 ,3), padding=1),
            torch.nn.PReLU(),
            
            torch.nn.Conv2d(in_channels=s, out_channels=d, kernel_size=(1 ,1), padding=0),
            torch.nn.PReLU(),

            torch.nn.ConvTranspose2d(in_channels=d, out_channels=1, kernel_size=(9 ,9), stride=(4, 4), padding=4, output_padding=3)
        )

inputs = (torch.randn(1, 1, 750, 750), )

macs = FlopCountAnalysis(model, inputs)

macs = macs.total()
flops = macs * 2

print(f'MACs = {macs / 1e+9:,} G')
print(f'FLOPs = {flops / 1e+9:,} G')
</code></pre>
<pre><code class="lang-auto">MACs = 7.011 G
FLOPs = 14.022 G
</code></pre>
<p>Just keep in mind if you set  <code>padding = 'same'</code> or <code>padding = 'valid'</code> in pytorch it gives you:</p>
<pre><code class="lang-auto">MACs = 2.5515 G
FLOPs = 5.103 G
</code></pre>
<p>which I think it is wrong.<br>
I calculate padding from fomula in <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" rel="noopener nofollow ugc">Conv2d</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html" rel="noopener nofollow ugc">ConvTranspose2d</a>.</p>
<h1><a name="tensorflow-3" class="anchor" href="#tensorflow-3"></a><strong>Tensorflow</strong></h1>
<p>According to the information provided in <a href="https://github.com/tensorflow/tensorflow/issues/32809#issuecomment-1065141109" rel="noopener nofollow ugc">github issue</a>, you can compute MACs and FLOPs in tensorflow using the following snippet code:</p>
<pre><code class="lang-auto">import tensorflow as tf
from tensorflow.python.profiler.model_analyzer import profile
from tensorflow.python.profiler.option_builder import ProfileOptionBuilder

def get_flops(model):
  forward_pass = tf.function(model.call, input_signature=[tf.TensorSpec(shape=(1,) + model.input_shape[1:])])
  graph_info = profile(forward_pass.get_concrete_function().graph, options=ProfileOptionBuilder.float_operation())
  flops = graph_info.total_float_ops
  return flops

model = tf.keras.applications.ConvNeXtTiny()

# model.compile(optimizer='adam', loss='bce', metrics=['accuracy'])

flops = get_flops(model)
macs = flops / 2
print(f"MACs: {macs / 1e+9:,} G")
print(f"FLOPs: {flops / 1e+9:,} G")
</code></pre>
<pre><code class="lang-auto">MACs: 4.3900329795 G
FLOPs: 8.780065959 G
</code></pre>
<p>Just keep in mind that the <a href="https://github.com/tensorflow/tensorflow/issues/32809#issuecomment-1555722328" rel="noopener nofollow ugc">source</a> computes <strong>MACs</strong> .</p>
<h2><a name="your-model-4" class="anchor" href="#your-model-4"></a><strong>Your model</strong></h2>
<pre><code class="lang-auto">d=56
s=12

model = tf.keras.Sequential([
    tf.keras.Input((750 ,750, 1)),

    tf.keras.layers.Conv2D(d, (5,5), padding='same'),
    tf.keras.layers.PReLU(),


    tf.keras.layers.Conv2D(s, (1,1), padding='valid'),
    tf.keras.layers.PReLU(),

    tf.keras.layers.Conv2D(s, (3,3), padding='same'),
    tf.keras.layers.PReLU(),

    tf.keras.layers.Conv2D(s, (3,3), padding='same'),
    tf.keras.layers.PReLU(),

    tf.keras.layers.Conv2D(s, (3,3), padding='same'),
    tf.keras.layers.PReLU(),

    tf.keras.layers.Conv2D(s, (3,3), padding='same'),
    tf.keras.layers.PReLU(),

    tf.keras.layers.Conv2D(d, (1,1), padding='same'),
    tf.keras.layers.PReLU(),

    tf.keras.layers.Conv2DTranspose(1 ,(9,9), strides=(4, 4),padding='same',output_padding = 3)
])

def get_flops(model):
  forward_pass = tf.function(model.call, input_signature=[tf.TensorSpec(shape=(1,) + model.input_shape[1:])])
  graph_info = profile(forward_pass.get_concrete_function().graph, options=ProfileOptionBuilder.float_operation())
  flops = graph_info.total_float_ops
  return flops

flops = get_flops(model)
macs = flops / 2
print(f"MACs: {macs / 1e+9:,} G")
print(f"FLOPs: {flops / 1e+9:,} G")
</code></pre>
<pre><code class="lang-auto">MACs: 7.257375 G
FLOPs: 14.51475 G
</code></pre>
<p><strong>In case</strong> your question is about that little difference between 2 frameworks, I think developers have to help us.</p>
<h1><a name="convnext-5" class="anchor" href="#convnext-5"></a><strong>ConvNeXt</strong></h1>
<p>If you refer to Figure 2 of <a href="https://openaccess.thecvf.com/content/CVPR2022/papers/Liu_A_ConvNet_for_the_2020s_CVPR_2022_paper.pdf" rel="noopener nofollow ugc">ConvNeXt</a> paper, you will notice that they have reported MACs instead of FLOPs for ConvNeXtTiny. ConvNeXt code: <a href="https://github.com/keras-team/keras/blob/v2.14.0/keras/applications/convnext.py" rel="noopener nofollow ugc">tensorflow</a> , <a href="https://github.com/facebookresearch/ConvNeXt" rel="noopener nofollow ugc">pytorch</a></p>
<p>I executed the code on colab.<br>
<strong>I kindly request that someone correct me if I am mistaken.</strong></p>