Repository: pytorch/ignite

Issue:
Make `MetricsLambda` update it's underlying metrics
## 🚀 Feature

As title.

The current state is that we cannot use `MetricsLambda` as a standard metric because it doesn't update/reset it's underlying metrics.
Adding this will expand the scope and usability of `MetricsLambda`

I've already an idea how to implement this based on our discussion https://github.com/pytorch/ignite/pull/1887#discussion_r610902667

So @vfdev-5 please assign the issue to me.




Discussion:
I'm not sure that LambdaMetric should update its related metrics. If so, what happen if the metrics are attached to an engine ? It could lead to multiple updates. I remember past discussions on that point and we stated that the update of metrics must be on user side.
@sdesrozis yes that's true. However, it means that our metrics arithmetics does not work with reset/update/compute API : 
```python
import torch
import ignite
print(ignite.__version__)

from ignite.metrics import Accuracy

err = (1.0 - Accuracy()) * 100.0

y_preds = torch.rand(4, 10)
y = torch.randint(0, 10, size=(4, ))

err.update((y_preds, y))
err.compute()
> NotComputableError
```

An idea could be to make `MetricsLambda` aware of the fact that it is attached or not and dispatch `update` according to that. What do you think ?
> @sdesrozis yes that's true. However, it means that our metrics arithmetics does not work with reset/update/compute API :
> 
> ```python
> import torch
> import ignite
> print(ignite.__version__)
> 
> from ignite.metrics import Accuracy
> 
> err = (1.0 - Accuracy()) * 100.0
> 
> y_preds = torch.rand(4, 10)
> y = torch.randint(0, 10, size=(4, ))
> 
> err.update((y_preds, y))
> err.compute()
> > NotComputableError
> ```
> 
> An idea could be to make `MetricsLambda` aware of the fact that it is attached or not and dispatch `update` according to that. What do you think ?

That can be an option. Another idea can be that we have a new subclass called `ComputableMetricsLambda` or something like this and it won't allow engines to be attached to it.
>  Another idea can be that we have a new subclass called ComputableMetricsLambda or something like this and it won't allow engines to be attached to it.

This can be ambiguous, I think. For example, `err = (1.0 - Accuracy()) * 100.0` would be `MetricsLambda` or `ComputableMetricsLambda` ? User can create a new metric like that and could use it in both scenarios (separately)...
I understand but I'm not sure...

Metrics `LambdaMetric` should check if its metrics would be already attached or not every update calls. Moreover, it means metrics would have to handle a state concerning attachment or not, and detachment.

A simple alternative would be to pass metrics to the report and not construct and attached internally... 

IMO a report is not a metric. It uses metrics. 
> A simple alternative would be to pass metrics to the report and not construct and attached internally...

For me, the main point of the issue here is to solve this issue : https://github.com/pytorch/ignite/issues/1921#issuecomment-817144804
```python
err = (1.0 - Accuracy()) * 100.0
```

> An idea could be to make `MetricsLambda` aware of the fact that it is attached or not and dispatch `update` according to that. What do you think ?

I need to write codes to go further on this
> For me, the main point of the issue here is to solve this issue : https://github.com/pytorch/ignite/issues/1921#issuecomment-817144804
> 
> ```python
> 
> err = (1.0 - Accuracy()) * 100.0
> 
> ```
 
It's a good use case. 

Here, we have one `LambdaMetric` per operator (minus and mult) defined by the tree op. Call `err.update(...)` means recursive checks on the tree where the leaf is `Accuracy()`. This metric is not attached so the `update(...)` should automatically called. 

It looks simple but error prone
```
acc = Accuracy()
err = (1.0 - acc) * 100.0

y_preds = torch.rand(4, 10)
y = torch.randint(0, 10, size=(4, ))

acc.update((y_preds, y))
err.update((y_preds, y))
err.compute()
```



Resolving patch:
diff --git a/ignite/metrics/metrics_lambda.py b/ignite/metrics/metrics_lambda.py
--- a/ignite/metrics/metrics_lambda.py
+++ b/ignite/metrics/metrics_lambda.py
@@ -15,9 +15,9 @@ class MetricsLambda(Metric):
     The result of the new metric is defined to be the result
     of applying the function to the result of argument metrics.
 
-    When update, this metric does not recursively update the metrics
+    When update, this metric recursively updates the metrics
     it depends on. When reset, all its dependency metrics would be
-    resetted. When attach, all its dependency metrics would be attached
+    resetted as well. When attach, all its dependency metrics would be attached
     automatically (but partially, e.g :meth:`~ignite.metrics.metric.Metric.is_attached()` will return False).
 
     Args:
@@ -71,6 +71,7 @@ def __init__(self, f: Callable, *args: Any, **kwargs: Any) -> None:
         self.args = args
         self.kwargs = kwargs
         self.engine = None  # type: Optional[Engine]
+        self._updated = False
         super(MetricsLambda, self).__init__(device="cpu")
 
     @reinit__is_reduced
@@ -78,13 +79,21 @@ def reset(self) -> None:
         for i in itertools.chain(self.args, self.kwargs.values()):
             if isinstance(i, Metric):
                 i.reset()
+        self._updated = False
 
     @reinit__is_reduced
     def update(self, output: Any) -> None:
-        # NB: this method does not recursively update dependency metrics,
-        # which might cause duplicate update issue. To update this metric,
-        # users should manually update its dependencies.
-        pass
+        if self.engine:
+            raise ValueError(
+                "MetricsLambda is already attached to an engine, "
+                "and MetricsLambda can't use update API while it's attached."
+            )
+
+        for i in itertools.chain(self.args, self.kwargs.values()):
+            if isinstance(i, Metric):
+                i.update(output)
+
+        self._updated = True
 
     def compute(self) -> Any:
         materialized = [_get_value_on_cpu(i) for i in self.args]
@@ -105,6 +114,10 @@ def _internal_attach(self, engine: Engine, usage: MetricUsage) -> None:
                     engine.add_event_handler(usage.ITERATION_COMPLETED, metric.iteration_completed)
 
     def attach(self, engine: Engine, name: str, usage: Union[str, MetricUsage] = EpochWise()) -> None:
+        if self._updated:
+            raise ValueError(
+                "The underlying metrics are already updated, can't attach while using reset/update/compute API."
+            )
         usage = self._check_usage(usage)
         # recursively attach all its dependencies (partially)
         self._internal_attach(engine, usage)


Test patch:
diff --git a/tests/ignite/metrics/test_metrics_lambda.py b/tests/ignite/metrics/test_metrics_lambda.py
--- a/tests/ignite/metrics/test_metrics_lambda.py
+++ b/tests/ignite/metrics/test_metrics_lambda.py
@@ -99,6 +99,124 @@ def fn(x, y, z, t):
     assert m2.list_ is None
 
 
+def test_metrics_lambda_update_and_attach_together():
+
+    y_pred = torch.randint(0, 2, size=(15, 10, 4)).float()
+    y = torch.randint(0, 2, size=(15, 10, 4)).long()
+
+    def update_fn(engine, batch):
+        y_pred, y = batch
+        return y_pred, y
+
+    engine = Engine(update_fn)
+
+    precision = Precision(average=False)
+    recall = Recall(average=False)
+
+    def Fbeta(r, p, beta):
+        return torch.mean((1 + beta ** 2) * p * r / (beta ** 2 * p + r)).item()
+
+    F1 = MetricsLambda(Fbeta, recall, precision, 1)
+
+    F1.attach(engine, "f1")
+    with pytest.raises(ValueError, match=r"MetricsLambda is already attached to an engine"):
+        F1.update((y_pred, y))
+
+    y_pred = torch.randint(0, 2, size=(15, 10, 4)).float()
+    y = torch.randint(0, 2, size=(15, 10, 4)).long()
+
+    F1 = MetricsLambda(Fbeta, recall, precision, 1)
+    F1.update((y_pred, y))
+
+    engine = Engine(update_fn)
+
+    with pytest.raises(ValueError, match=r"The underlying metrics are already updated"):
+        F1.attach(engine, "f1")
+
+    F1.reset()
+    F1.attach(engine, "f1")
+
+
+def test_metrics_lambda_update():
+
+    """
+    Test if the underlying metrics are updated
+    """
+    y_pred = torch.randint(0, 2, size=(15, 10, 4)).float()
+    y = torch.randint(0, 2, size=(15, 10, 4)).long()
+
+    precision = Precision(average=False)
+    recall = Recall(average=False)
+
+    def Fbeta(r, p, beta):
+        return torch.mean((1 + beta ** 2) * p * r / (beta ** 2 * p + r)).item()
+
+    F1 = MetricsLambda(Fbeta, recall, precision, 1)
+
+    F1.update((y_pred, y))
+
+    assert precision._updated
+    assert recall._updated
+
+    F1.reset()
+
+    assert not precision._updated
+    assert not recall._updated
+
+    """
+    Test multiple updates and if the inputs of
+    the underlying metrics are updated multiple times
+    """
+    y_pred1 = torch.randint(0, 2, size=(15,))
+    y1 = torch.randint(0, 2, size=(15,))
+
+    y_pred2 = torch.randint(0, 2, size=(15,))
+    y2 = torch.randint(0, 2, size=(15,))
+
+    F1.update((y_pred1, y1))
+    F1.update((y_pred2, y2))
+
+    # Compute true_positives and positives for precision
+    correct1 = y1 * y_pred1
+    all_positives1 = y_pred1.sum(dim=0)
+    if correct1.sum() == 0:
+        true_positives1 = torch.zeros_like(all_positives1)
+    else:
+        true_positives1 = correct1.sum(dim=0)
+
+    correct2 = y2 * y_pred2
+    all_positives2 = y_pred2.sum(dim=0)
+    if correct2.sum() == 0:
+        true_positives2 = torch.zeros_like(all_positives2)
+    else:
+        true_positives2 = correct2.sum(dim=0)
+
+    true_positives = true_positives1 + true_positives2
+    positives = all_positives1 + all_positives2
+
+    assert precision._type == "binary"
+    assert precision._true_positives == true_positives
+    assert precision._positives == positives
+
+    # Computing positivies for recall is different
+    positives1 = y1.sum(dim=0)
+    positives2 = y2.sum(dim=0)
+    positives = positives1 + positives2
+
+    assert recall._type == "binary"
+    assert recall._true_positives == true_positives
+    assert recall._positives == positives
+
+    """
+    Test compute
+    """
+    F1.reset()
+    F1.update((y_pred1, y1))
+    F1_metrics_lambda = F1.compute()
+    F1_sklearn = f1_score(y1.numpy(), y_pred1.numpy())
+    assert pytest.approx(F1_metrics_lambda) == F1_sklearn
+
+
 def test_integration():
     np.random.seed(1)