Repository: pytorch/ignite

Issue:
Evaluation/Metrics approach
This has been discussed a bit in other issues, but I wanted to make a dedicated issue for us to discuss this as I think it's very important we get this right.

## Background
Some previous discussions [here](https://github.com/pytorch/ignite/pull/54#issuecomment-363231705) and [here](https://github.com/pytorch/ignite/issues/72).

Throughout, I am going to use the motivating example of training a supervised model where you periodically want to compute some metrics against a validation set.

## Current Setup
Currently, in order to accomplish this, you need to do the following:

1. create an `Evaluator`
2. register the `Evaluate` handler to run the `Evaluator` on the validation set and store the predictions in the history
3. add another event handler to actually use this history to compute the metrics you care about
4. log/plot these metrics however you choose

In code, this looks something like this:

```python
model = ...
validation_loader = ...
trainer = ...
evaluator = create_supervised_evaluator(model, cuda=True)
trainer.add_event_handler(Events.EPOCH_COMPLETED, Evaluate(evaluator, validation_loader, epoch_interval=1))
@trainer.on(Events.EPOCH_COMPLETED)
def log(engine):
    print(engine.current_epoch, categorical_accuracy(evaluator.history))
```

### Pros
- keeps library code cleanly separated with minimal implicit dependencies
- user doesn't have to write much code

### Cons
- can be confusing what happens where
  - we have both an `Evaluator` and an `Evaluate` and yet neither one computes any sort of metrics
- there are a lot of ways this could go wrong
  - you have to understand the contract between what gets stored in the `Evaluator`'s history and the metrics functions
  - you have to make sure you attach the `Evaluate` handler and any logging handlers to the same event

## Goals
Evaluating a model is gonna be something that (essentially) everyone does so I think we need to have a good story here. Imo, we should make the supervised case be super easy while still making it possible for non-supervised cases. That being said, I think we want to accomplish this without removing flexibility and without adding a ton of code.

## Ideas
Working backward from what I would like the api to be, it might be nice if you could just do something like this:

```python
model = ...
validation_loader = ...
trainer = ...
@trainer.on(Events.EPOCH_COMPLETED)
def evaluate(engine):
  results = evaluate(model, {'acc': categorical_accuracy})
  # do something with those results
```

It'd be even nicer if I could do something like this:
```python
model = ...
validation_loader = ...
trainer = ...
trainer.add_event_handler(Events.EPOCH_COMPLETED, Evaluate(model, {'acc': categorical_accuracy}))
```

But without making assumptions about how users want to plot/log their evaluation results, this isn't possible. 

What do you all think? Anything here you take issue with? Any ideas on how we can best accomplish this? Do we need to make plotting/logging part of this discussion as well?

Discussion:

Patch:
diff --git a/ignite/evaluator.py b/ignite/evaluator.py
index 345acce0..249fe72d 100644
--- a/ignite/evaluator.py
+++ b/ignite/evaluator.py
@@ -12,7 +12,7 @@ class Evaluator(Engine):
         super(Evaluator, self).add_event_handler(event_name, handler, *args, **kwargs)
 
     def run(self, data):
-        state = State(dataloader=data)
+        state = State(dataloader=data, metrics={})
         self._fire_event(Events.STARTED, state)
         hours, mins, secs = self._run_once_on_dataset(state)
         self._logger.info("Evaluation Complete. Time taken: %02d:%02d:%02d", hours, mins, secs)
@@ -20,16 +20,17 @@ class Evaluator(Engine):
         return state
 
 
-def create_supervised_evaluator(model, cuda=False):
+def create_supervised_evaluator(model, metrics={}, cuda=False):
     """
     Factory function for creating an evaluator for supervised models
 
     Args:
         model (torch.nn.Module): the model to train
+        metrics (dict of str: Metric): a map of metric names to Metrics
         cuda (bool, optional): whether or not to transfer batch to GPU (default: False)
 
     Returns:
-        Trainer: a trainer instance with supervised inference function
+        Evaluator: a evaluator instance with supervised inference function
     """
     def _prepare_batch(batch):
         x, y = batch
@@ -43,4 +44,9 @@ def create_supervised_evaluator(model, cuda=False):
         y_pred = model(x)
         return y_pred.data.cpu(), y.data.cpu()
 
-    return Evaluator(_inference)
+    evaluator = Evaluator(_inference)
+
+    for name, metric in metrics.items():
+        metric.attach(evaluator, name)
+
+    return evaluator
diff --git a/ignite/exceptions.py b/ignite/exceptions.py
new file mode 100644
index 00000000..c8c0a94d
--- /dev/null
+++ b/ignite/exceptions.py
@@ -0,0 +1,4 @@
+class NotComputableError(RuntimeError):
+    """
+    Exception class to raise if Metric cannot be computed.
+    """
diff --git a/ignite/handlers/__init__.py b/ignite/handlers/__init__.py
index 8b142dd3..ee3f463c 100644
--- a/ignite/handlers/__init__.py
+++ b/ignite/handlers/__init__.py
@@ -1,1 +1,2 @@
+from .checkpoint import ModelCheckpoint
 from .timing import Timer
diff --git a/ignite/handlers/checkpoint.py b/ignite/handlers/checkpoint.py
new file mode 100644
index 00000000..2b2b8ba7
--- /dev/null
+++ b/ignite/handlers/checkpoint.py
@@ -0,0 +1,157 @@
+import os
+import tempfile
+
+import torch
+
+
+class ModelCheckpoint(object):
+    """ ModelCheckpoint handler can be used to periodically save objects to disk.
+
+    This handler accepts three arguments:
+        - an `ignite.engine.Engine` object
+        - an `ignite.engine.State` object, which will be passed to the
+            `score_function` (if it is provided)
+        - a `dict` mapping names (`str`) to objects that should be saved to disk.
+            See Notes and Examples for further details.
+
+    Args:
+        dirname (str):
+            Directory path where objects will be saved
+        filename_prefix (str):
+            Prefix for the filenames to which objects will be saved. See Notes
+            for more details.
+        save_interval (int, optional):
+            if not None, objects will be saved to disk every `save_interval` calls to the handler.
+            Exactly one of (`save_interval`, `score_function`) arguments must be provided.
+        score_function (Callable, optional):
+            if not None, it should be a function taking a 1single argument,
+            an `ignite.engine.State` object,
+            and return a score (`float`). Objects with highest scores will be retained.
+            Exactly one of (`save_interval`, `score_function`) arguments must be provided.
+        n_saved (int, optional):
+            Number of objects that should be kept on disk. Older files will be removed.
+        atomic (bool, optional):
+            If True, objects are serialized to a temporary file,
+            and then moved to final destination, so that files are
+            guaranteed to not be damaged (for example if exception occures during saving).
+        require_empty (bool, optional):
+            If True, will raise exception if there are any files starting with `filename_prefix`
+            in the directory 'dirname'
+        create_dir (bool, optional):
+            If True, will create directory 'dirname' if it doesnt exist.
+        exist_ok (bool, optional):
+            Passed to 'os.makedirs' call. Ignored if 'create_dir' is False.
+
+    Notes:
+          This handler expects three arguments: an `Engine` object,
+          a `State` object, and a `dict` mapping names to objects that should
+          be saved.
+          These names are used to specify filenames for saved objects.
+          Each filename has the following structure:
+          `{filename_prefix}_{name}_{step_number}.pth`.
+          Here, `filename_prefix` is the argument passed to the constructor,
+          `name` is the key in the aforementioned `dict`, and `step_number`
+          is incremented by `1` with every call to the handler.
+
+    Examples:
+        >>> import os
+        >>> from ignite.engine import Events
+        >>> from ignite.handlers import ModelCheckpoint
+        >>> from ignite.trainer import Trainer
+        >>> from torch import nn
+        >>> trainer = Trainer(lambda batch: None)
+        >>> handler = ModelCheckpoint('/tmp/models', 'myprefix', save_interval=2, n_saved=2, create_dir=True)
+        >>> model = nn.Linear(3, 3)
+        >>> trainer.add_event_handler(Events.EPOCH_COMPLETED, handler, {'mymodel': model})
+        >>> trainer.run([0], max_epochs=6)
+        >>> os.listdir('/tmp/models')
+        ['myprefix_mymodel_4.pth', 'myprefix_mymodel_6.pth']
+    """
+
+    def __init__(self, dirname, filename_prefix,
+                 save_interval=None, score_function=None,
+                 n_saved=1,
+                 atomic=True, require_empty=True,
+                 create_dir=True, exist_ok=False):
+
+        self._dirname = dirname
+        self._fname_prefix = filename_prefix
+        self._n_saved = n_saved
+        self._save_interval = save_interval
+        self._score_function = score_function
+        self._atomic = atomic
+        self._saved = []  # list of tuples (priority, saved_objects)
+        self._iteration = 0
+
+        if not (save_interval is None) ^ (score_function is None):
+            raise ValueError("Exactly one of `save_interval`, or `score_function` "
+                             "arguments must be provided.")
+
+        if create_dir:
+            exists = os.path.exists(dirname)
+            if exists and not exist_ok:
+                raise OSError("Directory {} already exists. Pass exist_ok=True to ignore this error.")
+            elif not exists:
+                os.makedirs(dirname)
+
+        if require_empty:
+            matched = [fname
+                       for fname in os.listdir(dirname)
+                       if fname.startswith(self._fname_prefix)]
+
+            if len(matched) > 0:
+                raise ValueError("Files prefixed with {} are already present "
+                                 "in the directory {}. If you want to use this "
+                                 "directory anyway, pass `require_empty=False`. "
+                                 "".format(filename_prefix, dirname))
+
+    def _save(self, obj, path):
+        if not self._atomic:
+            torch.save(obj, path)
+
+        else:
+            tmp = tempfile.NamedTemporaryFile(delete=False)
+
+            try:
+                torch.save(obj, tmp.file)
+
+            except BaseException:
+                tmp.close()
+                os.remove(tmp.name)
+                raise
+
+            else:
+                tmp.close()
+                os.rename(tmp.name, path)
+
+    def __call__(self, engine, state, to_save):
+        if len(to_save) == 0:
+            raise RuntimeError("No objects to checkpoint found.")
+
+        self._iteration += 1
+
+        if self._score_function is not None:
+            priority = self._score_function(state)
+
+        else:
+            priority = self._iteration
+            if (self._iteration % self._save_interval) != 0:
+                return
+
+        if (len(self._saved) < self._n_saved) or (self._saved[0][0] < priority):
+            saved_objs = []
+
+            for name, obj in to_save.items():
+                fname = '{}_{}_{}.pth'.format(self._fname_prefix, name, self._iteration)
+                path = os.path.join(self._dirname, fname)
+
+                self._save(obj=obj, path=path)
+                saved_objs.append(path)
+
+            self._saved.append((priority, saved_objs))
+            self._saved.sort(key=lambda item: item[0])
+
+        if len(self._saved) > self._n_saved:
+            _, paths = self._saved.pop(0)
+            for p in paths:
+                os.remove(p)
diff --git a/ignite/metrics/__init__.py b/ignite/metrics/__init__.py
new file mode 100644
index 00000000..3902c5de
--- /dev/null
+++ b/ignite/metrics/__init__.py
@@ -0,0 +1,3 @@
+from .categorical_accuracy import CategoricalAccuracy
+from .mean_squared_error import MeanSquaredError
+from .metric import Metric
diff --git a/ignite/metrics/categorical_accuracy.py b/ignite/metrics/categorical_accuracy.py
new file mode 100644
index 00000000..a08c0672
--- /dev/null
+++ b/ignite/metrics/categorical_accuracy.py
@@ -0,0 +1,29 @@
+from __future__ import division
+
+import torch
+
+from .metric import Metric
+from ignite.exceptions import NotComputableError
+
+
+class CategoricalAccuracy(Metric):
+    """
+    Calculates the categorical accuracy.
+
+    `update` must receive output of the form (y_pred, y).
+    """
+    def reset(self):
+        self._num_correct = 0
+        self._num_examples = 0
+
+    def update(self, output):
+        y_pred, y = output
+        indices = torch.max(y_pred, 1)[1]
+        correct = torch.eq(indices, y)
+        self._num_correct += torch.sum(correct)
+        self._num_examples += correct.shape[0]
+
+    def compute(self):
+        if self._num_examples == 0:
+            raise NotComputableError('CategoricalAccuracy must have at least one example before it can be computed')
+        return self._num_correct / self._num_examples
diff --git a/ignite/metrics/mean_squared_error.py b/ignite/metrics/mean_squared_error.py
new file mode 100644
index 00000000..98f9533c
--- /dev/null
+++ b/ignite/metrics/mean_squared_error.py
@@ -0,0 +1,28 @@
+from __future__ import division
+
+import torch
+
+from .metric import Metric
+from ignite.exceptions import NotComputableError
+
+
+class MeanSquaredError(Metric):
+    """
+    Calculates the mean squared error.
+
+    `update` must receive output of the form (y_pred, y).
+    """
+    def reset(self):
+        self._sum_of_squared_errors = 0.0
+        self._num_examples = 0
+
+    def update(self, output):
+        y_pred, y = output
+        squared_errors = torch.pow(y_pred - y.view_as(y_pred), 2)
+        self._sum_of_squared_errors += torch.sum(squared_errors)
+        self._num_examples += y.shape[0]
+
+    def compute(self):
+        if self._num_examples == 0:
+            raise NotComputableError('MeanSquaredError must have at least one example before it can be computed')
+        return self._sum_of_squared_errors / self._num_examples
diff --git a/ignite/metrics/metric.py b/ignite/metrics/metric.py
new file mode 100644
index 00000000..d4ea0da4
--- /dev/null
+++ b/ignite/metrics/metric.py
@@ -0,0 +1,66 @@
+from abc import ABCMeta, abstractmethod
+
+from ignite.engine import Events
+
+
+class Metric(object):
+    __metaclass__ = ABCMeta
+
+    """
+    Base class for all Metrics.
+
+    Metrics provide a way to compute various quantities of interest in an online
+    fashion without having to store the entire output history of a model.
+    """
+    def __init__(self):
+        self.reset()
+
+    @abstractmethod
+    def reset(self):
+        """
+        Resets the metric to to it's initial state.
+
+        This is called at the start of each evaluation run.
+        """
+        pass
+
+    @abstractmethod
+    def update(self, output):
+        """
+        Updates the metric's state using the passed batch output.
+
+        This is called once for each batch of each evaluation run.
+
+        Args:
+            output: the is the output from the evaluator's process function
+        """
+        pass
+
+    @abstractmethod
+    def compute(self):
+        """
+        Computes the metric based on it's accumulated state.
+
+        This is called at the end of each evaluation run.
+
+        Returns:
+            Any: the actual quantity of interest
+
+        Raises:
+            NotComputableError: raised when the metric cannot be computed
+        """
+        pass
+
+    def started(self, engine, state):
+        self.reset()
+
+    def iteration_completed(self, engine, state):
+        self.update(state.output)
+
+    def completed(self, engine, state, name):
+        state.metrics[name] = self.compute()
+
+    def attach(self, engine, name):
+        engine.add_event_handler(Events.STARTED, self.started)
+        engine.add_event_handler(Events.ITERATION_COMPLETED, self.iteration_completed)
+        engine.add_event_handler(Events.COMPLETED, self.completed, name)


Test patch:
diff --git a/tests/ignite/handlers/test_checkpoint.py b/tests/ignite/handlers/test_checkpoint.py
new file mode 100644
index 00000000..35a14b0b
--- /dev/null
+++ b/tests/ignite/handlers/test_checkpoint.py
@@ -0,0 +1,128 @@
+import os
+import tempfile
+
+import pytest
+import torch
+import shutil
+
+from ignite.engine import Events
+from ignite.handlers import ModelCheckpoint
+from ignite.trainer import Trainer
+
+_PREFIX = 'PREFIX'
+
+
+@pytest.fixture
+def dirname():
+    path = tempfile.mkdtemp()
+    yield path
+    shutil.rmtree(path)
+
+
+def test_args_validation(dirname):
+    existing = os.path.join(dirname, 'existing_dir')
+    nonempty = os.path.join(dirname, 'nonempty')
+
+    os.makedirs(existing)
+    os.makedirs(nonempty)
+
+    with open(os.path.join(nonempty, '{}_name_0.pth'.format(_PREFIX)), 'w'):
+        pass
+
+    # save_interval & score_func
+    with pytest.raises(ValueError):
+        h = ModelCheckpoint(existing, _PREFIX,
+                            create_dir=False)
+
+    with pytest.raises(OSError):
+        h = ModelCheckpoint(existing, _PREFIX, create_dir=True,
+                            save_interval=42)
+
+    with pytest.raises(ValueError):
+        h = ModelCheckpoint(nonempty, _PREFIX, exist_ok=True,
+                            save_interval=42)
+
+
+def test_simple_recovery(dirname):
+    h = ModelCheckpoint(dirname, _PREFIX, create_dir=False, save_interval=1)
+    h(None, None, {'obj': 42})
+
+    fname = os.path.join(dirname, '{}_{}_{}.pth'.format(_PREFIX, 'obj', 1))
+    assert torch.load(fname) == 42
+
+
+def test_atomic(dirname):
+    serializable = 42
+    non_serializable = (42, lambda _: 42)
+
+    def _test_existance(atomic, name, obj, expected):
+        h = ModelCheckpoint(dirname, _PREFIX,
+                            atomic=atomic,
+                            create_dir=False,
+                            require_empty=False,
+                            save_interval=1)
+
+        try:
+            h(None, None, {name: obj})
+        except:
+            pass
+
+        fname = os.path.join(dirname, '{}_{}_{}.pth'.format(_PREFIX, name, 1))
+        assert os.path.exists(fname) == expected
+
+    _test_existance(atomic=False, name='nonatomic_OK', obj=serializable, expected=True)
+    _test_existance(atomic=False, name='nonatomic_FAIL', obj=non_serializable, expected=True)
+
+    _test_existance(atomic=True, name='atomic_OK', obj=serializable, expected=True)
+    _test_existance(atomic=True, name='atomic_FAIL', obj=non_serializable, expected=False)
+
+
+def test_last_k(dirname):
+    h = ModelCheckpoint(dirname, _PREFIX, create_dir=False, n_saved=2, save_interval=2)
+    to_save = {'name': 42}
+
+    for _ in range(8):
+        h(None, None, to_save)
+
+    expected = ['{}_{}_{}.pth'.format(_PREFIX, 'name', i)
+                for i in [6, 8]]
+
+    assert sorted(os.listdir(dirname)) == expected
+
+
+def test_best_k(dirname):
+    scores = iter([1.0, -2., 3.0, -4.0])
+
+    def score_function(state):
+        return next(scores)
+
+    h = ModelCheckpoint(dirname, _PREFIX, create_dir=False,
+                        n_saved=2, score_function=score_function)
+
+    to_save = {'name': 42}
+    for _ in range(4):
+        h(None, None, to_save)
+
+    expected = ['{}_{}_{}.pth'.format(_PREFIX, 'name', i)
+                for i in [1, 3]]
+
+    assert sorted(os.listdir(dirname)) == expected
+
+
+def test_with_trainer(dirname):
+
+    def update_fn(batch):
+        pass
+
+    name = 'model'
+    trainer = Trainer(update_fn)
+    handler = ModelCheckpoint(dirname, _PREFIX, create_dir=False,
+                              n_saved=2, save_interval=1)
+
+    trainer.add_event_handler(Events.EPOCH_COMPLETED, handler, {name: 42})
+    trainer.run([0], max_epochs=4)
+
+    expected = ['{}_{}_{}.pth'.format(_PREFIX, name, i)
+                for i in [3, 4]]
+
+    assert sorted(os.listdir(dirname)) == expected
diff --git a/tests/ignite/metrics/test_categorical_accuracy.py b/tests/ignite/metrics/test_categorical_accuracy.py
new file mode 100644
index 00000000..ebd00594
--- /dev/null
+++ b/tests/ignite/metrics/test_categorical_accuracy.py
@@ -0,0 +1,25 @@
+from ignite.exceptions import NotComputableError
+from ignite.metrics import CategoricalAccuracy
+import pytest
+import torch
+
+
+def test_zero_div():
+    acc = CategoricalAccuracy()
+    with pytest.raises(NotComputableError):
+        acc.compute()
+
+
+def test_compute():
+    acc = CategoricalAccuracy()
+
+    y_pred = torch.eye(4)
+    y = torch.ones(4).type(torch.LongTensor)
+    acc.update((y_pred, y))
+    assert acc.compute() == 0.25
+
+    acc.reset()
+    y_pred = torch.eye(2)
+    y = torch.ones(2).type(torch.LongTensor)
+    acc.update((y_pred, y))
+    assert acc.compute() == 0.5
diff --git a/tests/ignite/metrics/test_mean_squared_error.py b/tests/ignite/metrics/test_mean_squared_error.py
new file mode 100644
index 00000000..3be6845f
--- /dev/null
+++ b/tests/ignite/metrics/test_mean_squared_error.py
@@ -0,0 +1,25 @@
+from ignite.exceptions import NotComputableError
+from ignite.metrics import MeanSquaredError
+import pytest
+import torch
+
+
+def test_zero_div():
+    mse = MeanSquaredError()
+    with pytest.raises(NotComputableError):
+        mse.compute()
+
+
+def test_compute():
+    mse = MeanSquaredError()
+
+    y_pred = torch.Tensor([[2.0], [-2.0]])
+    y = torch.zeros(2)
+    mse.update((y_pred, y))
+    assert mse.compute() == 4.0
+
+    mse.reset()
+    y_pred = torch.Tensor([[3.0], [-3.0]])
+    y = torch.zeros(2)
+    mse.update((y_pred, y))
+    assert mse.compute() == 9.0
diff --git a/tests/ignite/test_evaluator.py b/tests/ignite/test_evaluator.py
index ee455cb0..cf4f6172 100644
--- a/tests/ignite/test_evaluator.py
+++ b/tests/ignite/test_evaluator.py
@@ -5,6 +5,7 @@ from torch.nn import Linear
 
 from ignite.engine import Events, State
 from ignite.evaluator import Evaluator, create_supervised_evaluator
+from ignite.metrics import MeanSquaredError
 
 
 def test_returns_state():
@@ -121,3 +122,18 @@ def test_create_supervised():
 
     assert model.weight.data[0, 0] == approx(0.0)
     assert model.bias.data[0] == approx(0.0)
+
+
+def test_create_supervised_with_metrics():
+    model = Linear(1, 1)
+    model.weight.data.zero_()
+    model.bias.data.zero_()
+
+    evaluator = create_supervised_evaluator(model, metrics={'mse': MeanSquaredError()})
+
+    x = torch.FloatTensor([[1.0], [2.0]])
+    y = torch.FloatTensor([[3.0], [4.0]])
+    data = [(x, y)]
+
+    state = evaluator.run(data)
+    assert state.metrics['mse'] == 12.5