Repository: pytorch/ignite

Issue:
Discussion about ProgressBar and metrics display
Hello, I would like to discuss the feature in `contrib.handlers.ProgressBar` that displays metrics on each `Events.ITERATION_COMPLETED` as a postfix of the bar. For some context, please read https://github.com/pytorch/ignite/pull/295.

I think the `ProgressBar` and this feature are essential to the package, as it provides a concise way to have a nice training and evaluation experience (replacing the old way of spamming numbers in the console), and is appealing to beginners.

Here is some of the issues I found.

## 1. Non-`RunningAverage` metrics
### Problem
On the doc, the given example is
```python
pbar = ProgressBar()
pbar.attach(trainer, ['loss'])
```

The `metrics` parameter (here `['loss']`) works with `Metric`s that have been attached to the engine, **AND that rewired `self.computed` (which updates the `state.metrics` dict of the engine) on `Events.ITERATION_COMPLETED` instead of `Events.EPOCH_COMPLETED`**. The second part is unintuitive, and not documented. The current document suggests that this might work with all metrics, but this raises a `KeyError` when the dict has not been updated before the end of the first iteration.
Currently, this freature only works with `RunningAverage`.

The problem has been raised by @vfdev-5 in https://github.com/pytorch/ignite/pull/256, but I did not see the discussion go anywhere, I would like for it to continue.

### Proposed solutions

1. The documentation should be updated to reflect the fact that this feature will not work on all of the built-in metrics but `RunningAverage`.

2. The error message for the `KeyError` is `metrics not found in engine.state.metrics` and could be modified to hint for what the issue might be.

3. a Maybe the `Metric` class hierarchy should be updated, to distinguish metrics that update the engine state every epoch, and those that update it every iteration. The `ProgressBar` could then only accept the latter one. 

3. b Or maybe all of the current built-in metrics could update the state of the engine at each iteration (on `self.update` instead of on `self.completed`. This will add computation at each epoch. The progress bar would then work on all metrics.

## 2. Print the loss of a training engine
### Problem
Have a look at the doc example again :
```python
pbar = ProgressBar()
pbar.attach(trainer, ['loss'])
```
This suggests that we can print the training loss, that updates at each iteration.

However, we can't. When we are using a trainer created with `engine.create_supervised_trainer`, the `process_function` returns the loss for the processed batch. I think this is the most intuitive way to do it even for custom training engines.
The `Metric`s that can be attached are computed from the output of this function (`y_pred, y` for an evaluator for example), which may pass through a transform function. You see where I am going: the output of the trainer is already the loss itself, we do not need any further computation.

If we want to print the training loss, we have to attach a `Metric` that does nothing but moves the already computed loss into the `state.metrics` dict of the engine at every iteration. See https://github.com/pytorch/ignite/pull/295 for the `Identity` metric. This is not intuitive, we should be able to display the training loss out of the box.

### Proposed solutions

1. Built-in Identity metric, see https://github.com/pytorch/ignite/pull/295 for the code.

2. Modify `ProgressBar` to allow the display in the postfix the value of `engine.state.output`
    The `attach` method could then be 
    ```python
    attach(self, engine, metric_names=None, display_output=False)
    ```
    with the output looking like 
    `Epoch 1: [59/689]   9%|▊                 , output=3.54e-02 [00:04<00:50]`
3. Modify `ProgressBar` to allow the display in the postfix the value of any custom variables in the `engine.state` object, including the output.

---
Thanks for reading, sorry for the long issue and the english mistakes.


Discussion:
Hey @milesial, thanks for your input.

> Built-in Identity metric, see #295 for the code.

I agree that we should have a wrapper for the engine output to be saved as a metric, this would be the easiest and cleaner solution.

> Modify ProgressBar to allow the display in the postfix the value of engine.state.output

I think this is off the table because different engines can have different outputs, so it wouldn't generalize.

> Modify ProgressBar to allow the display in the postfix the value of any custom variables in the engine.state object, including the output.

this could also be interesting, but it may be tricky to generalize to all the possible cases. We had a discussion when I first implemented the `ProgressBar` about handling the output as well, but the code was quickly getting messy and confusing.

@vfdev-5 what's your opinion on these?
> I think this is off the table because different engines can have different outputs, so it wouldn't generalize.

I agree, one solution could be to ask for a function `output_to_str`, so the user is responsible for formatting the output. If no such function is passed, we can assume the returned variable is a float, string, ..., or has a `__repr__` method.
Concerning the issue, current implementation of `ProgressBar` actually works with other metrics if they are already computed. I agree that we need to improve the documentation on this.

Concerning the output and custom variables. What do you think about a solution as it is done for [`RunningAverage`](https://github.com/pytorch/ignite/blob/master/ignite/metrics/running_average.py#L9) ?

So a new API of `ProgressBar` can look like : 
```python
pbar = ProgressBar()
# current behaviour
pbar.attach(trainer, metric_names=['some_running_metric', ], output_transform=None)

# display transformed output
pbar.attach(trainer, metric_names=None, output_transform=lambda x: x)

# display custom variable
pbar.attach(trainer, metric_names=None, output_transform=None, custom_variables=['custom_var', ])

# progress bar only: default values
pbar.attach(trainer, metric_names=None, output_transform=None, custom_variables=None)
```

This is not a perfect solution if user has multiple outputs to display with a custom messages etc. But in this situation I wonder if user should not use `log_message` as all shown progress is not logged persistently and even removed when epoch ends.


I can make an option to persist the bar...
About the persistency, I was thinking about displaying the training loss for each batch, of course the loss of the last batch will be overwritten by the loss of the new batch. This allows us to look at the bar progression and see the evolution of the loss without getting spammed. At the end of the epoch, a handler can be used to print persistently the total average loss of the epoch.
> About the persistency, I was thinking about displaying the  training loss for each batch, of course the loss of the last batch will  be overwritten by the loss of the new batch.

@milesial Actually, why you do not want to use a running average with loss at each batch ? As all discussion we started from this...




I want to see the non-smoothed value of the current loss (this is why I proposed alpha=0 in https://github.com/pytorch/ignite/pull/295)
So in the API, should  `output_transform` return a printable object that the `ProgressBar` can use ? What if the returned object is already printable : a float, string, ... ? passing `lambda x: x` seems unnecessary.
In general case the output of an engine is not a single scalar, tensor etc, we have no limitation on the its form. That's why we need to transform it into something meaningful. On the other hand if `output_transform` is not `None` in the API it means it is activated and need to be displayed, otherwise output display is omitted. 
Okay, it would be nice to be able to pass `output_transform=True`, and it would treat it as `output_transform=lambda x: x`. But then the name `output_transform` is strange, we could rename it to `display_output`.

So we can pass the following into the `display_output` parameter :
- `None` or `False`: do not display output
- a callable : pass through it before displaying
- `True` : display without going through a callable (or applying `lambda x: x`, same effect)

In my mind, in the case of `create_supervised_trainer`, it would be cleaner to simply be able to :
```python
pbar.attach(trainer, display_output=True)
```
With the ability to pass callable to process different outputs, as you proposed.
We have already introduced `output_transform` before in Metrics and I think we should keep same API for similar stuff here and not introduce something different instead. And keep thing simple.

-    `None`: do not display output
-    a callable : pass through it before displaying. 



Okay, I can live with that.
But for the `custom_variables` we assume they are all printable ? A list of tuples (var_name, output_transform) seems complicated.
Yes, we can explicitly detail in the doc that progress bar can display only printable variables.
> But for the `custom_variables` we assume they are all printable ? A list of tuples (var_name, output_transform) seems complicated.

you would actually have to transform it into a dictionary `{name: value}` for it to make sense

> we could rename it to `display_output`

This doesn't generalize, you are only thinking about the `create_supervised_trainer` case. If you look at the DCGAN example you'll see the `step` function output it is actually a dictionary, for example.

Resolving patch:
diff --git a/ignite/contrib/handlers/tqdm_logger.py b/ignite/contrib/handlers/tqdm_logger.py
--- a/ignite/contrib/handlers/tqdm_logger.py
+++ b/ignite/contrib/handlers/tqdm_logger.py
@@ -10,47 +10,83 @@ class ProgressBar:
     """
     TQDM progress bar handler to log training progress and computed metrics.
 
+    Args:
+        persist (bool, optional): set to ``True`` to persist the progress bar after completion (default = ``False``)
+
     Examples:
 
-        Create a progress bar that shows you some metrics as they are computed,
-        by simply attaching the progress bar object to your engine.
+        Simple progress bar
+
+        .. code-block:: python
+
+            trainer = create_supervised_trainer(model, optimizer, loss)
+
+            pbar = ProgressBar()
+            pbar.attach(trainer)
+
+        Attach metrics that already have been computed at `ITERATION_COMPLETED` (such as `RunningAverage`)
 
         .. code-block:: python
 
+            trainer = create_supervised_trainer(model, optimizer, loss)
+
+            RunningAverage(output_transform=lambda x: x).attach(trainer, 'loss')
+
             pbar = ProgressBar()
             pbar.attach(trainer, ['loss'])
 
+        Directly attach the engine's output
+
+        .. code-block:: python
+
+            trainer = create_supervised_trainer(model, optimizer, loss)
+
+            pbar = ProgressBar()
+            pbar.attach(trainer, output_transform=lambda x: {'loss': x})
+
     Note:
         When adding attaching the progress bar to an engine, it is recommend that you replace
         every print operation in the engine's handlers triggered every iteration with
         ``pbar.log_message`` to guarantee the correct format of the stdout.
     """
 
-    def __init__(self):
+    def __init__(self, persist=False):
         self.pbar = None
+        self.persist = persist
 
     def _reset(self, engine):
         self.pbar = tqdm(
             total=len(engine.state.dataloader),
-            leave=False,
+            leave=self.persist,
             bar_format='{desc}[{n_fmt}/{total_fmt}] {percentage:3.0f}%|{bar}{postfix} [{elapsed}<{remaining}]')
 
     def _close(self, engine):
         self.pbar.close()
         self.pbar = None
 
-    def _update(self, engine, metric_names=None):
+    def _update(self, engine, metric_names=None, output_transform=None):
         if self.pbar is None:
             self._reset(engine)
 
         self.pbar.set_description('Epoch {}'.format(engine.state.epoch))
 
+        metrics = {}
         if metric_names is not None:
             if not all(metric in engine.state.metrics for metric in metric_names):
                 self._close(engine)
                 raise KeyError("metrics not found in engine.state.metrics")
 
-            metrics = {name: '{:.2e}'.format(engine.state.metrics[name]) for name in metric_names}
+            metrics.update({name: '{:.2e}'.format(engine.state.metrics[name]) for name in metric_names})
+
+        if output_transform is not None:
+            output_dict = output_transform(engine.state.output)
+
+            if not isinstance(output_dict, dict):
+                output_dict = {"output": output_dict}
+
+            metrics.update({name: '{:.2e}'.format(value) for name, value in output_dict.items()})
+
+        if metrics:
             self.pbar.set_postfix(**metrics)
 
         self.pbar.update()
@@ -65,16 +101,23 @@ def log_message(message):
         """
         tqdm.write(message)
 
-    def attach(self, engine, metric_names=None):
+    def attach(self, engine, metric_names=None, output_transform=None):
         """
         Attaches the progress bar to an engine object
 
         Args:
             engine (Engine): engine object
-            metric_names (list): (Optional) list of the metrics names to log as the bar progresses
+            metric_names (list, optional): list of the metrics names to log as the bar progresses
+            output_transform (Callable, optional): a function to select what you want to print from the engine's
+                output. This function may return either a dictionary with entries in the format of ``{name: value}``,
+                or a single scalar, which will be displayed with the default name `output`.
         """
         if metric_names is not None and not isinstance(metric_names, list):
             raise TypeError("metric_names should be a list, got {} instead".format(type(metric_names)))
 
+        if output_transform is not None and not callable(output_transform):
+            raise TypeError("output_transform should be a function, got {} instead"
+                            .format(type(output_transform)))
+
         engine.add_event_handler(Events.EPOCH_COMPLETED, self._close)
-        engine.add_event_handler(Events.ITERATION_COMPLETED, self._update, metric_names)
+        engine.add_event_handler(Events.ITERATION_COMPLETED, self._update, metric_names, output_transform)


Test patch:
diff --git a/tests/ignite/contrib/handlers/test_pbar.py b/tests/ignite/contrib/handlers/test_pbar.py
--- a/tests/ignite/contrib/handlers/test_pbar.py
+++ b/tests/ignite/contrib/handlers/test_pbar.py
@@ -9,8 +9,9 @@
 
 
 def update_fn(engine, batch):
-    engine.state.metrics['a'] = 1
-    return None
+    a = 1
+    engine.state.metrics['a'] = a
+    return a
 
 
 def test_pbar(capsys):
@@ -86,3 +87,47 @@ def test_pbar_no_metric_names(capsys):
     actual = err[-1]
     expected = u'Epoch 2: [1/2]  50%|█████      [00:00<00:00]'
     assert actual == expected
+
+
+def test_pbar_with_output(capsys):
+    n_epochs = 2
+    loader = [1, 2]
+    engine = Engine(update_fn)
+
+    pbar = ProgressBar()
+    pbar.attach(engine, output_transform=lambda x: {'a': x})
+
+    engine.run(loader, max_epochs=n_epochs)
+
+    captured = capsys.readouterr()
+    err = captured.err.split('\r')
+    err = list(map(lambda x: x.strip(), err))
+    err = list(filter(None, err))
+    expected = u'Epoch 2: [1/2]  50%|█████     , a=1.00e+00 [00:00<00:00]'
+    assert err[-1] == expected
+
+
+def test_pbar_fail_with_non_callable_transform():
+    engine = Engine(update_fn)
+    pbar = ProgressBar()
+
+    with pytest.raises(TypeError):
+        pbar.attach(engine, output_transform=1)
+
+
+def test_pbar_with_scalar_output(capsys):
+    n_epochs = 2
+    loader = [1, 2]
+    engine = Engine(update_fn)
+
+    pbar = ProgressBar()
+    pbar.attach(engine, output_transform=lambda x: x)
+
+    engine.run(loader, max_epochs=n_epochs)
+
+    captured = capsys.readouterr()
+    err = captured.err.split('\r')
+    err = list(map(lambda x: x.strip(), err))
+    err = list(filter(None, err))
+    expected = u'Epoch 2: [1/2]  50%|█████     , output=1.00e+00 [00:00<00:00]'
+    assert err[-1] == expected