Clarification of FLOps/MACs in model descriptions

Guillaume_Jeanneret: <p>Hi there!</p>
<p>I noticed that the FLOps reported in the <a href="https://pytorch.org/vision/stable/models.html#table-of-all-available-classification-weights" rel="noopener nofollow ugc">torchvision library</a> are different from those computed with the torch profiler. So I computed the FLOps for several architectures from the torchvision library using meta’s <a href="https://github.com/facebookresearch/fvcore" rel="noopener nofollow ugc">fvcore library</a> and the official torch profiler:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>architecture</th>
<th>reported</th>
<th>fvcore</th>
<th>torch profiler</th>
</tr>
</thead>
<tbody>
<tr>
<td>AlexNet</td>
<td>0.71</td>
<td>0.71</td>
<td>1.43</td>
</tr>
<tr>
<td>ResNet 50</td>
<td>4.09</td>
<td>4.11</td>
<td>8.18</td>
</tr>
<tr>
<td>DenseNet 121</td>
<td>2.83</td>
<td>2.87</td>
<td>5.67</td>
</tr>
<tr>
<td>Swin B</td>
<td>15.43</td>
<td>15.47</td>
<td>30.88</td>
</tr>
<tr>
<td>MaxViT T</td>
<td>5.56</td>
<td>5.61</td>
<td>11.13</td>
</tr>
<tr>
<td>ViT-B 16</td>
<td>17.56</td>
<td>16.87</td>
<td>33.70</td>
</tr>
</tbody>
</table>
</div><p>From these results, the torch profiler calculates the actual FLOps, while the fvcore library calculates the <a href="https://github.com/facebookresearch/fvcore/issues/69" rel="noopener nofollow ugc">MACs</a> (FLOps ~= 2MACs). However, the descriptions of the torchvision models in <a href="https://pytorch.org/vision/stable/models.html#table-of-all-available-classification-weights" rel="noopener nofollow ugc">this link</a> show the MACs and not the FLOps. Is there any way to clarify this in the official documentation?</p>
<p>Also, I noticed that the MACs of the ViT-B differ from the reported value ($\sim 5 %$). Do you have any idea why?</p>
<p>Thank you very much!</p>
<p>Code snippet to calculate the FLOps with torch 2.1 and cuda 11.8:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torchvision.models as models
from fvcore.nn import FlopCountAnalysis, flop_count_table, flop_count_str

torch.set_grad_enabled(False)
models_to_load = ['alexnet', 'resnet50', 'densenet121', 'swin_b', 'maxvit_t', 'vit_b_16']
input = torch.randn(1, 3, 224, 224, device='cuda')

class AutoCastModel(torch.nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model

    @torch.autocast('cuda')  # to use the traditional attention instead of the flash attention
    def forward(self, x):
        return self.model(x)

for m in models_to_load:
    print('=' * 79)
    print(m)
    print('=' * 79)
    model = AutoCastModel(getattr(models, m)())
    model.cuda()
    model.eval()

    flop = FlopCountAnalysis(model, input)
    print(flop_count_table(flop, max_depth=0))
    with torch.profiler.profile(with_flops=True) as p, torch.autocast('cuda'):
        _ = model(input)
    # print(p.key_averages().table(sort_by="flops", row_limit=5))
    print('{:.2f} GFLOPS (torch profile)'.format(sum(k.flops for k in p.key_averages()) / 1e9))
</code></pre>

qsj287068067: <p>I also observed the same phenomenon with my customized model. The FLOPs counted by <code>torch.profiler.profile</code> is approximately 2 times of that counted by <code>mmengine.analysis.get_model_complexity_info()</code> (<a href="https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/model_analysis.md" rel="noopener nofollow ugc">mmengine doc</a>).</p>
<p>I suspect this is caused by the unclear definition of <code>FLOPs</code>. <code>torch.profiler.profile</code> counts multiply-add as two operations according with <a href="https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html#gpu-arch" rel="noopener nofollow ugc">nvidia’s document</a>:</p>
<blockquote>
<p>Each multiply-add comprises two operations, thus one would multiply the throughput in the table by 2 to get FLOP counts per clock.</p>
</blockquote>
<p>While <code>mmengine</code> following <code>fvcore</code> to calculate multiply-add as one operation.</p>