Repository: pytorch/ignite

Issue:
[Feature Request] "Removable" event handlers.
_I'm happy to open a PR for this immediately if the feature seems useful._


Ignite's event interface provides both add and remove operations however the "remove" operation requires a reference to the original event handler. This makes temporary or one-off event handlers somewhat tricky to implement.

Pytorch's existing hook interface provides a solution to this issue by returning a "RemovableHandle" handle from `register_hook`, holding a weakref to the target hook to support later removal.

Ignite's `add_event_handler` should implement a similar interface, returning a "RemovableEventHandle" rather than `None`. For example, in an attrs-style syntax:

```python
@attr.s(auto_attribs=True)
class RemovableEventHandle:
    """Removable weak reference to an added event handler."""


    event_name: str
    handler: weakref.ref = attr.ib(converter=_as_weakref)
    engine: weakref.ref = attr.ib(converter=_as_weakref)

    def remove(self):
        """Remove handler from engine."""
        handler = self.handler()
        engine: Engine = self.engine()

        if handler and engine and engine.has_event_handler(handler):
            engine.remove_event_handler(handler, self.event_name)

    def __enter__(self):  # noqa: D
        return self

    def __exit__(self, type, value, tb):  # noqa: D
        self.remove()
```

Would support features like:

```python
engine = Engine(process_function)

def print_epoch(engine):
    print("Epoch: {}".format(engine.state.epoch))

# Train with a temporary event handler.
with engine.add_event_handler(Events.EPOCH_COMPLETED, print_epoch):
    engine.run(dataloader, 10)
```


Discussion:
@asford thanks for the feature request, interesting observation about missing event handle! 
In my use-cases, such handle was event handler itself, such that I could remove the handler.

Providing a temporary event handler seems interesting idea, could you please provide more details for its use-cases ?



I've implemented and used this feature to support single-iteration diagnostic handlers connected to an ignite engine. As a concrete example, I'm debugging model training via a handler that accumulates parameter gradients for a single iteration, logs gradient histograms to tensorboard after iteration completion, and then self-detaches from the engine via the removable handlers. This allows me to log gradient histograms on every n-th iteration in a relatively cleanly.

As a skeleton "single-shot" handler:

```python
from contextlib import ContextStack

@attr.s
class LogGradientHandler:
    
    summary_writer = attr.ib()
    _handlers : ContextStack = attr.ib(factory=ContextStack, init=False)
    ... gradient monitoring state ...
    
    def for_iteration(self, engine, model):
        for name, param in dict(model.named_parameters()).items():
            parameter_hook = ...attach backward hook...
            self._handlers.enter_context(parameter_hook)
        
        self._handlers.enter_context(
            self.engine.add_event_handler(
                Events.ITERATION_COMPLETED, self._log_and_detach))

    def _log_and_detach(self, engine):
        ... summary_writer.add_histogram for stored_gradient in logged values ...
        # Remove backward hooks from parameters, unregister event.
        self._handlers.close()

model = ... create model ...
summary_writer = ... setup tensorboard ...
engine = ... setup engine ...

@engine.on(Events.ITERATION_STARTED)
def maybe_log_grads(self, engine):
    """ Log parameter gradient every 10th batch."""
    if engine.state.iteration % 10:
        LogGradientHandler(summary_writer).for_iteration(engine, model)

So if I understand correctly this attach-remove behaviour is needed for executing a handler once n iterations ?
If it is the case, there is also another contrib possibility to perform this using [custom events](https://pytorch.org/ignite/contrib/handlers.html#module-ignite.contrib.handlers.custom_events).
I wonder whether this could help in your situation ?

Another option how to use such self-detaching context manager could be with an evaluator on train and test datasets with specific handlers...
Yes, there are multiple ways of implementing the equivalent behavior for single-shot handlers. I've just found this method to be low-overhead and relatively easy to keep tidy.

As you point out, I'm also using this feature to "reuse" an evaluator engine across multiple datasets with different logging configurations.

I think the context manger support offered by this feature is the key to increased usability.
> I think the context manger support offered by this feature is the key to increased usability.

@asford I agree with that ! If you wish to send a PR, it would be awesome :)
IMO, the class `RemovableEventHandle` from your example, looks good. Probably, we could integrate it without `attrib` as a new dependency... What do yout think ?
> Probably, we could integrate it without `attrib` as a new dependency... What do yout think ?

Absolutely, I'll open a PR with a vanilla implementation shortly.

Resolving patch:
diff --git a/ignite/engine/engine.py b/ignite/engine/engine.py
--- a/ignite/engine/engine.py
+++ b/ignite/engine/engine.py
@@ -4,6 +4,7 @@
 import time
 from collections import defaultdict
 from enum import Enum
+import weakref
 
 from ignite._utils import _to_hours_mins_secs
 
@@ -48,6 +49,58 @@ def get_event_attrib_value(self, event_name):
         return getattr(self, State.event_to_attr[event_name])
 
 
+class RemovableEventHandle(object):
+    """A weakref handle to remove a registered event.
+
+    A handle that may be used to remove a registered event handler via the
+    remove method, with-statement, or context manager protocol. Returned from
+    :meth:`~ignite.engine.Engine.add_event_handler`.
+
+
+    Args:
+        event_name: Registered event name.
+        handler: Registered event handler, stored as weakref.
+        engine: Target engine, stored as weakref.
+
+    Example usage:
+
+    .. code-block:: python
+
+        engine = Engine()
+
+        def print_epoch(engine):
+            print("Epoch: {}".format(engine.state.epoch))
+
+        with engine.add_event_handler(Events.EPOCH_COMPLETED, print_epoch):
+            # print_epoch handler registered for a single run
+            engine.run(data)
+
+        # print_epoch handler is now unregistered
+    """
+
+    def __init__(self, event_name, handler, engine):
+        self.event_name = event_name
+        self.handler = weakref.ref(handler)
+        self.engine = weakref.ref(engine)
+
+    def remove(self):
+        """Remove handler from engine."""
+        handler = self.handler()
+        engine = self.engine()
+
+        if handler is None or engine is None:
+            return
+
+        if engine.has_event_handler(handler, self.event_name):
+            engine.remove_event_handler(handler, self.event_name)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, type, value, tb):
+        self.remove()
+
+
 class Engine(object):
     """Runs a given process_function over each batch of a dataset, emitting events as it goes.
 
@@ -164,6 +217,9 @@ def add_event_handler(self, event_name, handler, *args, **kwargs):
               Note that other arguments can be passed to the handler in addition to the `*args` and  `**kwargs`
               passed here, for example during :attr:`~ignite.engine.Events.EXCEPTION_RAISED`.
 
+        Returns:
+            :class:`~ignite.engine.RemovableEventHandler`, which can be used to remove the handler.
+
         Example usage:
 
         .. code-block:: python
@@ -186,6 +242,8 @@ def print_epoch(engine):
         self._event_handlers[event_name].append((handler, args, kwargs))
         self._logger.debug("added handler for event %s.", event_name)
 
+        return RemovableEventHandle(event_name, handler, self)
+
     def has_event_handler(self, handler, event_name=None):
         """Check if the specified event has the specified handler.
 


Test patch:
diff --git a/tests/ignite/engine/test_engine.py b/tests/ignite/engine/test_engine.py
--- a/tests/ignite/engine/test_engine.py
+++ b/tests/ignite/engine/test_engine.py
@@ -1,5 +1,6 @@
 from __future__ import division
 from enum import Enum
+import gc
 
 import pytest
 from mock import call, MagicMock, Mock
@@ -130,6 +131,85 @@ def test_adding_multiple_event_handlers():
         handler.assert_called_once_with(engine)
 
 
+def test_event_removable_handle():
+
+    # Removable handle removes event from engine.
+    engine = DummyEngine()
+    handler = MagicMock()
+
+    removable_handle = engine.add_event_handler(Events.STARTED, handler)
+    assert engine.has_event_handler(handler, Events.STARTED)
+
+    engine.run(1)
+    handler.assert_called_once_with(engine)
+
+    removable_handle.remove()
+    assert not engine.has_event_handler(handler, Events.STARTED)
+
+    # Second engine pass does not fire handle again.
+    engine.run(1)
+    handler.assert_called_once_with(engine)
+
+    # Removable handle can be used as a context manager
+    handler = MagicMock()
+
+    with engine.add_event_handler(Events.STARTED, handler):
+        assert engine.has_event_handler(handler, Events.STARTED)
+        engine.run(1)
+
+    assert not engine.has_event_handler(handler, Events.STARTED)
+    handler.assert_called_once_with(engine)
+
+    engine.run(1)
+    handler.assert_called_once_with(engine)
+
+    # Removeable handle only effects a single event registration
+    handler = MagicMock()
+
+    with engine.add_event_handler(Events.STARTED, handler):
+        with engine.add_event_handler(Events.COMPLETED, handler):
+            assert engine.has_event_handler(handler, Events.STARTED)
+            assert engine.has_event_handler(handler, Events.COMPLETED)
+        assert engine.has_event_handler(handler, Events.STARTED)
+        assert not engine.has_event_handler(handler, Events.COMPLETED)
+    assert not engine.has_event_handler(handler, Events.STARTED)
+    assert not engine.has_event_handler(handler, Events.COMPLETED)
+
+    # Removeable handle is re-enter and re-exitable
+
+    handler = MagicMock()
+
+    remove = engine.add_event_handler(Events.STARTED, handler)
+
+    with remove:
+        with remove:
+            assert engine.has_event_handler(handler, Events.STARTED)
+        assert not engine.has_event_handler(handler, Events.STARTED)
+    assert not engine.has_event_handler(handler, Events.STARTED)
+
+    # Removeable handle is a weakref, does not keep engine or event alive
+    def _add_in_closure():
+        _engine = DummyEngine()
+
+        def _handler(_):
+            pass
+
+        _handle = _engine.add_event_handler(Events.STARTED, _handler)
+        assert _handle.engine() is _engine
+        assert _handle.handler() is _handler
+
+        return _handle
+
+    removable_handle = _add_in_closure()
+
+    # gc.collect, resolving reference cycles in engine/state
+    # required to ensure object deletion in python2
+    gc.collect()
+
+    assert removable_handle.engine() is None
+    assert removable_handle.handler() is None
+
+
 def test_has_event_handler():
     engine = DummyEngine()
     handlers = [MagicMock(), MagicMock()]