Repository: ContinualAI/avalanche

Issue:
Add MAC metric
It would be nice to have also a [MAC ](https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation) metric. It's difficult to compute it in native pytorch for every possible layer, but it would be a nice, hardware independed metric.

Any idea on how to do this easily in PyTorch?


Discussion:
A simple implementation can ignore the cost of activation functions and only compute the cost of feedforward and convolutional layers. Then you have to manage all the cases for convolutional layers. 

The biggest problem is that for convolutional layers you need to know the input size, and therefore the layer's order.
@AntonioCarta Maybe we can create a script that given a pytorch model:

1. Convert every conv layer into a custom conv layer which also counts MAC for it (on-the-fly during the forward pass) and saves them in a common attribute of the model object (let's say a dict).
2. Run the model on the first mini-batch of the chosen benchmark and return MAC statistics

What do you think?

Yes, that may be a solution. It should also be possible with forward hooks without subclassing.
I can try to implement it.
@AntonioCarta Is this issue solved?
Not yet. If you want you can work on this @akshitac8.
@akshitac8 are you working on this? It's not clear to me since I see that I am still assigned to the issue. Please give me a confirmation and assign the issue to yourself if you are taking it.
Otherwise I will work on this.
I am sorry for not giving you a clear sign on it. I had a medical emergency but now I have started working on this bust not clear just MAC metric is needed or both this issue and #60 are related? 
if you can clear me on this and then you can assign me the issue :)
@AntonioCarta any news on this? Should we disassign this issue?
Sorry, I forgot this issue was still assigned to me. I can do it (with the limitations discussed above), give me about a week of time to complete it.

Resolving patch:
diff --git a/avalanche/evaluation/metrics.py b/avalanche/evaluation/metrics.py
--- a/avalanche/evaluation/metrics.py
+++ b/avalanche/evaluation/metrics.py
@@ -37,6 +37,48 @@
 import time
 
 
+class MAC:
+    """
+        Multiply-and-accumulate metric. Approximately measure the computational
+        cost of a model in a hardware-independent way by computing the number
+        of multiplications. Currently supports only Linear or Conv2d modules.
+        Other operations are ignored.
+    """
+    def __init__(self):
+        self.hooks = []
+        self._compute_cost = 0
+
+    def compute(self, model, dummy_input):
+        for mod in model.modules():
+            if self.is_recognized_module(mod):
+                def foo(a, b, c):
+                    return self.update_compute_cost(a, b, c)
+                handle = mod.register_forward_hook(foo)
+                self.hooks.append(handle)
+
+        self._compute_cost = 0
+        model(dummy_input)  # trigger forward hooks
+
+        for handle in self.hooks:
+            handle.remove()
+        self.hooks = []
+        return self._compute_cost
+
+    def update_compute_cost(self, module, input, output):
+        modname = module.__class__.__name__
+        if modname == 'Linear':
+            self._compute_cost += input[0].shape[1] * output.shape[1]
+        elif modname == 'Conv2d':
+            n, cout, hout, wout = output.shape  # Batch, Channels, Height, Width
+            ksize = module.kernel_size[0] * module.kernel_size[1]
+            self._compute_cost += cout * hout * wout * (ksize)
+        print(self._compute_cost)
+
+    def is_recognized_module(self, mod):
+        modname = mod.__class__.__name__
+        return modname == 'Linear' or modname == 'Conv2d'
+
+
 class GPUUsage:
     """
         GPU usage metric measured as average usage percentage over time.


Test patch:
diff --git a/tests/test_metrics.py b/tests/test_metrics.py
new file mode 100644
--- /dev/null
+++ b/tests/test_metrics.py
@@ -0,0 +1,40 @@
+""" Metrics Tests"""
+
+import unittest
+import torch
+from torch import nn
+
+from avalanche.evaluation.metrics import MAC
+
+
+class MultiTaskTests(unittest.TestCase):
+    def test_ff_model(self):
+        xn, hn, yn = 50, 100, 10
+
+        model = nn.Sequential(
+            nn.Linear(xn, hn),  # 5'000 mul
+            nn.ReLU(),
+            nn.Linear(hn, hn),  # 10'000 mul
+            nn.ReLU(),
+            nn.Linear(hn, yn)  # 1'000 mul
+        )  # 16'000 mul
+        dummy = torch.randn(32, xn)
+        met = MAC()
+        self.assertEqual(16000, met.compute(model, dummy))
+
+    def test_cnn_model(self):
+        xn, hn, yn = 50, 100, 10
+        model = nn.Sequential(
+            nn.Conv2d(1, 10, 4),  # 353'440 mul
+            nn.ReLU(),
+            nn.Flatten(),
+            nn.Linear(22090, hn),  # 2'209'000 mul
+            nn.Linear(hn, yn)  # 1'000 mul
+        )  # 2'563'440 mul
+        dummy = torch.randn(32, 1, xn, xn)
+        met = MAC()
+        print(2563440, met.compute(model, dummy))
+
+
+if __name__ == '__main__':
+    unittest.main()