Repository: pytorch/ignite

Issue:
[Feature Request] More general metrics
I find the metrics to be a bit limited as one might want to pass additional options (even tensors) to the loss.
For instance in recurrent models with different sequence lengths, one would use a mask to avoid counting errors on padded time steps.
The mask is necessary in the loss to know which outputs to use in the final averaging/ loss.
[Feature Request] More general metrics
I find the metrics to be a bit limited as one might want to pass additional options (even tensors) to the loss.
For instance in recurrent models with different sequence lengths, one would use a mask to avoid counting errors on padded time steps.
The mask is necessary in the loss to know which outputs to use in the final averaging/ loss.


Discussion:
Could you maybe give an example using code or pseudo-code?
@AntoinePrv could you please give an example on this issue ? 

If this becomes stale, maybe we can close this...
Sorry, I've postponed answering this for far too long...

Ideally we would want `ignite.metric.Loss` to be able to wrap any loss function from PyTorch, right?
Many loss functions have a `weights` attribute (tensor) to scale the different element (over the batch for instance).

Although the argument can be set statically before passing `loss_fn` to the `Loss`, this cannot be used dynamically (`weights` to depend on batch) with the current class (at least not nicely).
This is because of [this line](https://github.com/pytorch/ignite/blob/c71a3b04d2514f27b401d11a848115bebbae19d8/ignite/metrics/loss.py#L23): eventually only two arguments are extracted out of the output.

Something like the following might be an idea:
```python
args, kwargs = output
average_loss = self._loss_fn(*args, **kwargs)
```

(Note: this is also the case with the masks I was mentioning in the first comment, but we can stick to the example of `weights` as the point is the same)
@AntoinePrv coming back on what you say about `weights`. Following the [cross entropy docs](https://pytorch.org/docs/stable/nn.html#torch.nn.functional.cross_entropy), `weights`  rescales classes and not samples... 
If I understand correctly you would like to be able to apply weights coming from batch generator ?

Yes, I would like to be able to pass any argument (also by name) to the `loss_fn`.
If `Loss` is to be general enough, shouldn't it be able to wrap arbitrary arguments ?

Am I making sense? Or maybe my case is specific enough that I should write my own `Metric` for that purpose...
@AntoinePrv we discussed a similar point [here](https://github.com/pytorch/ignite/pull/112#issuecomment-373838309). Yes, I would say your remark totally make sense, just for instance, we have findout if we could cover this with our vanilla `Loss` or create somelthing like `LossWithArgs` ... Let me think about it. 

You would change the `output_transform` default to `lambda x: (x, {})` so that the evaluator still returns `y_pred, y`.

Note that using that loss would still require rewriting a custom evaluator to change the output.
@AntoinePrv a hacky solution can be something similar:
```python
import torch.nn.functional as F

def loss_fn(y_pred, target):
    y, kwargs = target    
    return F.cross_entropy(y_pred, y, **kwargs)

metrics = {
    'loss': Loss(loss_fn, output_transform=lambda y_pred, y, kwargs: (y_pred, (y, kwargs)))
}
```
Sure, I also made my own solution. I just wanted to know if you're interested in making this change.
I think @AntoinePrv's comments are valid, especially for per-batch `weights` and other args. If I understand the proposed solution correctly it will be backwards compatible?
Partially, however, it's hard to say because all of the tests involve:
```python
loss.update((y_pred, y))
```
I could succeed the tests by replacing it with
```python
loss.update(loss.update(((y_pred, y), {})))
```

I rely on changing the default argument to the constructor from `output_transform=lambda x: x` to `output_transform=lambda x: (x, {})`, which means it will not be backward compatible for any user having used that input.

__Note__:
I was too greedy with `args, kwargs = output` because `y` is used a few line after for:
```python
self._sum += average_loss.item() * y.shape[0]
self._num_examples += y.shape[0]
```
A sufficient change could be `(y_pred, y), kwargs = output`
I agree that this is a completely valid point and something we should address. If we want to be backwards compatible, I think the easiest thing to do is to change [this](https://github.com/pytorch/ignite/blob/master/ignite/metrics/loss.py#L23) to something roughly like this:

```python
y_pred, y = output[0], output[1]
kwargs = output[2] if len(output) > 2 else {}
```
lets get this into 0.1.1
Could you maybe give an example using code or pseudo-code?
@AntoinePrv could you please give an example on this issue ? 

If this becomes stale, maybe we can close this...
Sorry, I've postponed answering this for far too long...

Ideally we would want `ignite.metric.Loss` to be able to wrap any loss function from PyTorch, right?
Many loss functions have a `weights` attribute (tensor) to scale the different element (over the batch for instance).

Although the argument can be set statically before passing `loss_fn` to the `Loss`, this cannot be used dynamically (`weights` to depend on batch) with the current class (at least not nicely).
This is because of [this line](https://github.com/pytorch/ignite/blob/c71a3b04d2514f27b401d11a848115bebbae19d8/ignite/metrics/loss.py#L23): eventually only two arguments are extracted out of the output.

Something like the following might be an idea:
```python
args, kwargs = output
average_loss = self._loss_fn(*args, **kwargs)
```

(Note: this is also the case with the masks I was mentioning in the first comment, but we can stick to the example of `weights` as the point is the same)
@AntoinePrv coming back on what you say about `weights`. Following the [cross entropy docs](https://pytorch.org/docs/stable/nn.html#torch.nn.functional.cross_entropy), `weights`  rescales classes and not samples... 
If I understand correctly you would like to be able to apply weights coming from batch generator ?

Yes, I would like to be able to pass any argument (also by name) to the `loss_fn`.
If `Loss` is to be general enough, shouldn't it be able to wrap arbitrary arguments ?

Am I making sense? Or maybe my case is specific enough that I should write my own `Metric` for that purpose...
@AntoinePrv we discussed a similar point [here](https://github.com/pytorch/ignite/pull/112#issuecomment-373838309). Yes, I would say your remark totally make sense, just for instance, we have findout if we could cover this with our vanilla `Loss` or create somelthing like `LossWithArgs` ... Let me think about it. 

You would change the `output_transform` default to `lambda x: (x, {})` so that the evaluator still returns `y_pred, y`.

Note that using that loss would still require rewriting a custom evaluator to change the output.
@AntoinePrv a hacky solution can be something similar:
```python
import torch.nn.functional as F

def loss_fn(y_pred, target):
    y, kwargs = target    
    return F.cross_entropy(y_pred, y, **kwargs)

metrics = {
    'loss': Loss(loss_fn, output_transform=lambda y_pred, y, kwargs: (y_pred, (y, kwargs)))
}
```
Sure, I also made my own solution. I just wanted to know if you're interested in making this change.
I think @AntoinePrv's comments are valid, especially for per-batch `weights` and other args. If I understand the proposed solution correctly it will be backwards compatible?
Partially, however, it's hard to say because all of the tests involve:
```python
loss.update((y_pred, y))
```
I could succeed the tests by replacing it with
```python
loss.update(loss.update(((y_pred, y), {})))
```

I rely on changing the default argument to the constructor from `output_transform=lambda x: x` to `output_transform=lambda x: (x, {})`, which means it will not be backward compatible for any user having used that input.

__Note__:
I was too greedy with `args, kwargs = output` because `y` is used a few line after for:
```python
self._sum += average_loss.item() * y.shape[0]
self._num_examples += y.shape[0]
```
A sufficient change could be `(y_pred, y), kwargs = output`
I agree that this is a completely valid point and something we should address. If we want to be backwards compatible, I think the easiest thing to do is to change [this](https://github.com/pytorch/ignite/blob/master/ignite/metrics/loss.py#L23) to something roughly like this:

```python
y_pred, y = output[0], output[1]
kwargs = output[2] if len(output) > 2 else {}
```
lets get this into 0.1.1

Resolving patch:
diff --git a/ignite/metrics/loss.py b/ignite/metrics/loss.py
--- a/ignite/metrics/loss.py
+++ b/ignite/metrics/loss.py
@@ -8,9 +8,21 @@ class Loss(Metric):
     """
     Calculates the average loss according to the passed loss_fn.
 
-    - `loss_fn` must return the average loss over all observations in the batch.
-    - `update` must receive output of the form `(y_pred, y)`.
+    Args:
+        loss_fn (callable): a callable taking a prediction tensor, a target
+            tensor, optionally other arguments, and returns the average loss
+            over all observations in the batch.
+        output_transform (callable): a callable that is used to transform the
+            :class:`ignite.engine.Engine`'s `process_function`'s output into the
+            form expected by the metric.
+            This can be useful if, for example, you have a multi-output model and
+            you want to compute the metric with respect to one of the outputs.
+            The output is is expected to be a tuple (prediction, target) or
+            (prediction, target, kwargs) where kwargs is a dictionary of extra
+            keywords arguments.
+
     """
+
     def __init__(self, loss_fn, output_transform=lambda x: x):
         super(Loss, self).__init__(output_transform)
         self._loss_fn = loss_fn
@@ -20,8 +32,12 @@ def reset(self):
         self._num_examples = 0
 
     def update(self, output):
-        y_pred, y = output
-        average_loss = self._loss_fn(y_pred, y)
+        if len(output) == 2:
+            y_pred, y = output
+            kwargs = {}
+        else:
+            y_pred, y, kwargs = output
+        average_loss = self._loss_fn(y_pred, y, **kwargs)
         assert len(average_loss.shape) == 0, '`loss_fn` did not return the average loss'
         self._sum += average_loss.item() * y.shape[0]
         self._num_examples += y.shape[0]


Test patch:
diff --git a/tests/ignite/metrics/test_loss.py b/tests/ignite/metrics/test_loss.py
--- a/tests/ignite/metrics/test_loss.py
+++ b/tests/ignite/metrics/test_loss.py
@@ -50,6 +50,15 @@ def test_non_averaging_loss():
         loss.update((y_pred, y))
 
 
+def test_kwargs_loss():
+    loss = Loss(nll_loss)
+
+    y_pred = torch.Tensor([[0.1, 0.4, 0.5], [0.1, 0.7, 0.2]]).log()
+    y = torch.LongTensor([2, 2])
+    loss.update((y_pred, y, {"weight": torch.Tensor([0, 0, 0])}))
+    assert_almost_equal(loss.compute(), 0)
+
+
 def test_reset():
     loss = Loss(nll_loss)