Repository: pytorch/ignite

Issue:
Loss automatically detaching inputs breaks some workflows
## 🐛 Bug description

https://github.com/pytorch/ignite/blob/master/ignite/metrics/loss.py#L60

Before 0.4.3, the `y_pred` and `y` were passed without calling detach. Now because of detach a number of usecases are no longer possible.

An example is: https://docs.gpytorch.ai/en/v1.3.1/marginal_log_likelihoods.html#variationalelbo

The output of `model(train_x)` is an object which does not support detach. This leads to a crash when passing the `mll` function to Loss as a `loss_fn`.

Another example is https://github.com/y0ast/deterministic-uncertainty-quantification/blob/master/train_duq_cifar.py#L153

The loss is dependent on the gradient, which means that by calling detach the loss cannot be computed anymore.

I have been trying to work around it, but I can't figure out a nice way. https://pytorch.org/ignite/metrics.html#ignite.metrics.Average is not input size aware so it cannot correctly compute the average of the loss.


Discussion:
I see this exact point was discussed in https://github.com/pytorch/ignite/pull/1238#issuecomment-679166910 by @vfdev-5 and @n2cholas 

It seems to me that it's very common (and recommended from pytorch) to use `torch.no_grad():` during evaluation (which is when `Loss` is most useful, I use `Average` otherwise).

So the argument of making it easier for the user is there, but maybe less important than initial assumed?
@y0ast thanks for reporting that ! 

> Now because of detach a number of usecases are no longer possible.

Indeed, this should be fixed !

I wonder if we do not call detach on y_pred for Loss metric could make sense, what do you think ?
Naively, I'd say we should not continue building the graph for computing the metric... Probably, doing `y_pred.clone().detach()` could help ?

On the other hand, yes, maybe we can move `torch.no_grad` from `Metric.iteration_completed` to `Metric.update` and its derived classes:

https://github.com/pytorch/ignite/blob/0ea6de5f2d716bc8bebe238010bba2b168dd7b2e/ignite/metrics/metric.py#L274-L275


@y0ast I do not quite understand actually why VariationalELBO fails to call detach on the output of the model ?

> The loss is dependent on the gradient, which means that by calling detach the loss cannot be computed anymore.

I agree with that so cloning and detaching could work in this case, right ?
> Naively, I'd say we should not continue building the graph for computing the metric... Probably, doing y_pred.clone().detach() could help ?

Will still break https://github.com/y0ast/deterministic-uncertainty-quantification/blob/master/train_duq_cifar.py#L167 because the `loss_fn` actually needs a `y_pred` with gradient.

However that operation itself does not need to build the graph, so maybe your 
> On the other hand, yes, maybe we can move torch.no_grad from Metric.iteration_completed to Metric.update and its derived classes:

Would work! I am not 100% sure how it interacts with torch.autograd.grad.

Why not call `detach` on the **output** of `loss_fn`?

> I do not quite understand actually why VariationalELBO fails to call detach on the output of the model ?

```
        y_pred = model(x)
        elbo = -elbo_fn(y_pred, y)
```
elbo_fn is of type `VariationalELBO`
`y_pred` is of type `MultitaskMultivariateNormal` and `y` is just a `torch.Tensor, so if it's called as `elbo = -elbo_fn(y_pred.detach(), y.detach())` inside Loss, then `y_pred` needs to support detach.

Resolving patch:
diff --git a/ignite/metrics/loss.py b/ignite/metrics/loss.py
--- a/ignite/metrics/loss.py
+++ b/ignite/metrics/loss.py
@@ -57,7 +57,7 @@ def update(self, output: Sequence[Union[torch.Tensor, Dict]]) -> None:
             kwargs = {}  # type: Dict
         else:
             y_pred, y, kwargs = cast(Tuple[torch.Tensor, torch.Tensor, Dict], output)
-        average_loss = self._loss_fn(y_pred.detach(), y.detach(), **kwargs)
+        average_loss = self._loss_fn(y_pred, y, **kwargs).detach()
 
         if len(average_loss.shape) != 0:
             raise ValueError("loss_fn did not return the average loss.")


Test patch: