Repository: Project-MONAI/MONAI

Issue:
training stats event-handler (for printing training losses)



Discussion:
Hi @yanchengnv and @wyli ,

I found a similar example in @ericspod 's notebook example:
**@trainer.on(Events.EPOCH_COMPLETED)
def log_training_loss(engine):
    print("Epoch", engine.state.epoch, "Loss:", engine.state.output)**

About this task, do you mean something like it?
Thanks.

We would something a bit more generalized with optional outputs for end of iteration, end of step, end of train, etc. We ought to be logging to the engine's log object instead of printing to stdout, we can add and printing handler to that log object to do both of course.
Hi @ericspod ,

Thanks for your suggestion about general StatsLogger.
Intead of engine.log, can we put all the useful outputs to **engine.state.metrics**?
We can also get "iteration", "epoch", "max_epochs", "epoch_length", etc. from engine.state.
I think engine.state is designed to be an unified API to store useful information for all kinds of event-handlers.
If we aligned on this direction, I can try to make a PR based on engine.state.
Thanks.

Sorry for jumping in into your conversation, I just would like to help with that and make more clear on what is provided out-of-the-box in ignite for that:
- [TensorboardLogger](https://pytorch.org/ignite/contrib/handlers.html#ignite.contrib.handlers.tensorboard_logger.TensorboardLogger) if you would like to write any output parameters (e.g. batch losses), computed metrics, optimizer params etc.
- [VisdomLogger](https://pytorch.org/ignite/contrib/handlers.html#ignite.contrib.handlers.visdom_logger.VisdomLogger), same as TensorboardLogger but for visdom.
- [ProgressBar](https://pytorch.org/ignite/contrib/handlers.html#ignite.contrib.handlers.tqdm_logger.ProgressBar) is a tqdm bar for displaying the progress and showing some values of interest.

In my experience, using experiment tracking systems like MLflow or Polyaxon, we can either log to the system via their api (and ignite's wrappers like `MLflowLogger` or `PolyaxonLogger`), write events to TensorBoard or simply print values to stdout and this is automatically written to a log file. The first and the second approaches are obviously more interesting if we would like to compare different runs etc.

HTH
Hi @vfdev-5 ,

Thanks very much for your detailed sharing!
I will take a deep dive into your examples.

And @wyli @ericspod @yanchengnv ,

About the usage of Ignite, have we aligned to use only Ignite official code or both **official code** and the **3rd contrib code**?
Thanks.
@vfdev-5 @Nic-Ma  I've used the log file for logging just messages and such,  the SessionSaver class in ptproto creates a new directory in a given parent directory for every new run and sends the log to a file there along with the checkpoints and saved networks. My subclasses of Engine add extra fields to the state and we could add more things to it, I would think that metrics should only be the output from metric handlers and shouldn't have anything else. Returning to the idea of the session handling if we're saving the whole engine state (or everything without large tensors) then these other things we add will get saved as well.
@vfdev-5 One thing to mention is that tqdm doesn't play well with Jupyterlab for some reason, I believe it's a known bug. I had written a super primitive text progress bar that works, I don't know if we collectively want to investigate anything else. I really like doing things through Jupyter a lot, so stuff that doesn't rely on tensorboard/visdom is what I would prefer. 
There is a `tqdm_notebook` that works quite well: https://pypi.org/project/tqdm/#ipython-jupyter-integration
@fepegar I think that does have issues with Jupyterlab, Jupyter notebook vanilla I think is fine. I don't why but they're different.
Yes, I've had trouble before on JupyterLab. But I think installing the widgets extension solves it: https://ipywidgets.readthedocs.io/en/latest/user_install.html#installing-the-jupyterlab-extension
@ericspod I'm also using jupyterlab for development, it provides a cool environment for research/prototyping/testing etc.

> so stuff that doesn't rely on tensorboard/visdom is what I would prefer.

However, how do you plan to run and then organize and compare various trainings for the same task ? 
@fepegar I thought I had tried that and it didn't fix the issue, maybe it didn't load correct for me? I'll try again. 

@vfdev-5 That is something that I wasn't doing in a great way so definitely we should be targeting ways of supporting lab and tensorboard/visdom.
Yan - Setup time to discuss design choice here.

Resolving patch:
diff --git a/examples/unet_segmentation_3d.py b/examples/unet_segmentation_3d.py
--- a/examples/unet_segmentation_3d.py
+++ b/examples/unet_segmentation_3d.py
@@ -13,6 +13,7 @@
 import sys
 import tempfile
 from glob import glob
+import logging
 
 import nibabel as nib
 import numpy as np
@@ -26,6 +27,8 @@
 from monai import application, networks, utils
 from monai.data.readers import NiftiDataset
 from monai.data.transforms import (AddChannel, Rescale, ToTensor, UniformRandomPatch)
+from monai.application.handlers.stats_handler import StatsHandler
+from monai.networks.metrics.mean_dice import MeanDice
 
 # assumes the framework is found here, change as necessary
 sys.path.append("..")
@@ -96,7 +99,7 @@ def create_test_image_3d(height, width, depth, num_objs=12, rad_max=30, noise_ma
 loss = networks.losses.DiceLoss(do_sigmoid=True)
 opt = torch.optim.Adam(net.parameters(), lr)
 
-train_epochs = 30
+train_epochs = 3
 
 
 def _loss_fn(i, j):
@@ -106,11 +109,18 @@ def _loss_fn(i, j):
 device = torch.device("cuda:0")
 
 trainer = create_supervised_trainer(net, opt, _loss_fn, device, False,
-                                    output_transform=lambda x, y, y_pred, loss: [y_pred, loss.item()])
+                                    output_transform=lambda x, y, y_pred, loss: [y_pred, loss.item(), y])
 
 checkpoint_handler = ModelCheckpoint('./', 'net', n_saved=10, require_empty=False)
 trainer.add_event_handler(event_name=Events.EPOCH_COMPLETED, handler=checkpoint_handler, to_save={'net': net})
 
+dice_metric = MeanDice(add_sigmoid=True, output_transform=lambda output: (output[0][0], output[2]))
+dice_metric.attach(trainer, "Training Dice")
+
+logging.basicConfig(stream=sys.stdout, level=logging.INFO)
+stats_logger = StatsHandler()
+stats_logger.attach(trainer)
+
 
 @trainer.on(Events.EPOCH_COMPLETED)
 def log_training_loss(engine):
@@ -140,7 +150,7 @@ def log_training_loss(engine):
     third_label_tensor = torch.where(engine.state.batch[1][2] > 0, ones, engine.state.batch[1][2])
     utils.img2tensorboardutils.add_animated_gif(writer, "third_label_final_batch", third_label_tensor, 64,
                                                 255, engine.state.epoch)
-    print("Epoch", engine.state.epoch, "Loss:", engine.state.output[1])
+    engine.logger.info("Epoch[%s] Loss: %s", engine.state.epoch, engine.state.output[1])
 
 
 loader = DataLoader(ds, batch_size=20, num_workers=8, pin_memory=torch.cuda.is_available())
diff --git a/monai/application/handlers/stats_handler.py b/monai/application/handlers/stats_handler.py
new file mode 100644
--- /dev/null
+++ b/monai/application/handlers/stats_handler.py
@@ -0,0 +1,146 @@
+# Copyright 2020 MONAI Consortium
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from ignite.engine import Engine, Events
+
+KEY_VAL_FORMAT = '{}: {:.4f} '
+
+
+class StatsHandler(object):
+    """StatsHandler defines a set of Ignite Event-handlers for all the log printing logics.
+    It's can be used for any Ignite Engine(trainer, validator and evaluator).
+    And it can support logging for epoch level and iteration level with pre-defined StatsLoggers.
+    By default, this class logs the dictionary of `engine.state.metrics`.
+    """
+
+    def __init__(self,
+                 epoch_print_logger=None,
+                 iteration_print_logger=None,
+                 name=None):
+        """
+        Args:
+            epoch_print_logger (Callable): customized callable printer for epoch level logging.
+                                            must accept parameter "engine", use default printer if None.
+            iteration_print_logger (Callable): custimized callable printer for iteration level logging.
+                                            must accept parameter "engine", use default printer if None.
+            name (str): identifier of logging.logger to use, defaulting to `engine.logger`.
+        """
+
+        self.epoch_print_logger = epoch_print_logger
+        self.iteration_print_logger = iteration_print_logger
+
+        self.logger = None if name is None else logging.getLogger(name)
+
+
+    def attach(self, engine: Engine):
+        """Register a set of Ignite Event-Handlers to a specified Ignite engine.
+
+        Args:
+            engine (ignite.engine): Ignite Engine, it can be a trainer, validator or evaluator.
+
+        """
+        if self.logger is None:
+            self.logger = engine.logger
+        if not engine.has_event_handler(self.iteration_completed, Events.ITERATION_COMPLETED):
+            engine.add_event_handler(Events.ITERATION_COMPLETED, self.iteration_completed)
+        if not engine.has_event_handler(self.epoch_completed, Events.EPOCH_COMPLETED):
+            engine.add_event_handler(Events.EPOCH_COMPLETED, self.epoch_completed)
+        if not engine.has_event_handler(self.exception_raised, Events.EXCEPTION_RAISED):
+            engine.add_event_handler(Events.EXCEPTION_RAISED, self.exception_raised)
+
+    def epoch_completed(self, engine: Engine):
+        """handler for train or validation/evaluation epoch completed Event.
+        Print epoch level log, default values are from ignite state.metrics dict.
+
+        Args:
+            engine (ignite.engine): Ignite Engine, it can be a trainer, validator or evaluator.
+
+        """
+        if self.epoch_print_logger is not None:
+            self.epoch_print_logger(engine)
+        else:
+            self._default_epoch_print(engine)
+
+    def iteration_completed(self, engine: Engine):
+        """handler for train or validation/evaluation iteration completed Event.
+        Print iteration level log, default values are from ignite state.logs dict.
+
+        Args:
+            engine (ignite.engine): Ignite Engine, it can be a trainer, validator or evaluator.
+
+        """
+        if self.iteration_print_logger is not None:
+            self.iteration_print_logger(engine)
+        else:
+            self._default_iteration_print(engine)
+
+    def exception_raised(self, engine: Engine, e):
+        """handler for train or validation/evaluation exception raised Event.
+        Print the exception information and traceback.
+
+        Args:
+            engine (ignite.engine): Ignite Engine, it can be a trainer, validator or evaluator.
+            e (Exception): the exception catched in Ignite during engine.run().
+
+        """
+        self.logger.exception('Exception: {}'.format(e))
+        # import traceback
+        # traceback.print_exc()
+
+    def _default_epoch_print(self, engine: Engine):
+        """Execute epoch level log operation based on Ignite engine.state data.
+        print the values from ignite state.metrics dict.
+
+        Args:
+            engine (ignite.engine): Ignite Engine, it can be a trainer, validator or evaluator.
+
+        """
+        prints_dict = engine.state.metrics
+        if not prints_dict:
+            return
+        current_epoch = engine.state.epoch
+
+        out_str = "Epoch[{}] Metrics -- ".format(current_epoch)
+        for name in sorted(prints_dict):
+            value = prints_dict[name]
+            out_str += KEY_VAL_FORMAT.format(name, value)
+
+        self.logger.info(out_str)
+
+    def _default_iteration_print(self, engine: Engine):
+        """Execute iteration log operation based on Ignite engine.state data.
+        print the values from ignite state.logs dict.
+
+        Args:
+            engine (ignite.engine): Ignite Engine, it can be a trainer, validator or evaluator.
+
+        """
+        prints_dict = engine.state.metrics
+        if not prints_dict:
+            return
+        num_iterations = engine.state.epoch_length
+        current_iteration = (engine.state.iteration - 1) % num_iterations + 1
+        current_epoch = engine.state.epoch
+        num_epochs = engine.state.max_epochs
+
+        out_str = "Epoch: {}/{}, Iter: {}/{} -- ".format(
+            current_epoch,
+            num_epochs,
+            current_iteration,
+            num_iterations)
+
+        for name in sorted(prints_dict):
+            value = prints_dict[name]
+            out_str += KEY_VAL_FORMAT.format(name, value)
+
+        self.logger.info(out_str)


Test patch: