Repository: pytorch/ignite

Issue:
Improve or New time profiling utility 
## 🚀 Feature

Currently, ignite provides only a basic time profiling tool : https://pytorch.org/ignite/master/contrib/handlers.html#ignite.contrib.handlers.time_profilers.BasicTimeProfiler

Which can print something like 
```
----------------------------------------------------
| Time profiling stats (in seconds):                 |
 ----------------------------------------------------
total  |  min/index  |  max/index  |  mean  |  std

Processing function:
157.46292 | 0.01452/1501 | 0.26905/0 | 0.07730 | 0.01258

Dataflow:
6.11384 | 0.00008/1935 | 0.28461/1551 | 0.00300 | 0.02693

Event handlers:
2.82721

- Events.STARTED: []
0.00000

- Events.EPOCH_STARTED: []
0.00006 | 0.00000/0 | 0.00000/17 | 0.00000 | 0.00000

- Events.ITERATION_STARTED: ['PiecewiseLinear']
0.03482 | 0.00001/188 | 0.00018/679 | 0.00002 | 0.00001

- Events.ITERATION_COMPLETED: ['TerminateOnNan']
0.20037 | 0.00006/866 | 0.00089/1943 | 0.00010 | 0.00003

- Events.EPOCH_COMPLETED: ['empty_cuda_cache', 'training.<locals>.log_elapsed_time', ]
2.57860 | 0.11529/0 | 0.14977/13 | 0.12893 | 0.00790

- Events.COMPLETED: []
not yet triggered
```

What would be nice to have:
- more compact representation (maybe inspire from https://pytorch.org/docs/stable/autograd.html#torch.autograd.profiler.profile)
- profile handlers and not events. As we are looking for bottlenecks it would be helpful to know which functions take time and not on which event.
- better representation of data prep time vs training time




Discussion:
Hi @vfdev-5, I would love to work on this. Could you please assign it to me?
@harsh8398 sure ! Thanks !
Hello @vfdev-5, Sorry for lagging on this. So, I have some questions:

> profile handlers and not events. As we are looking for bottlenecks it would be helpful to know which functions take time and not on which event.

We want to add profiling for event handlers attached to engine/events instead of events? Essentially this would be break-down of time taken by handlers attached to events if I understand correctly.

> better representation of data prep time vs training time

Is this something we can achieve based on events? For example, aggregate diff between GET_BATCH_STARTED and GET_BATCH_COMPLETED?
Hi @harsh8398 

> We want to add profiling for event handlers attached to engine/events instead of events? Essentially this would be break-down of time taken by handlers attached to events if I understand correctly.

Yes, something like that. From the user's point of view, showing
```
- Events.EPOCH_COMPLETED: ['empty_cuda_cache', 'training.<locals>.log_elapsed_time', ]
2.57860 | 0.11529/0 | 0.14977/13 | 0.12893 | 0.00790
```
that handlers like `empty_cuda_cache` and `training.<locals>.log_elapsed_time` have taken together 0.12893 seconds on average does not tell who takes that time. As we can have one slow and another fast.
Naively, i'd list all handlers for all events and measure time for the handlers and display a table with handlers and time. Maybe something like that (but properly aligned):
```
-------
| name | total (s) | mean (s) | std(s) |
empty_cuda_cache (EPOCH_COMPLETED)                                |   2.57860 | 0.12893 | 0.00790
training.<locals>.log_elapsed_time (EPOCH_COMPLETED)      |  2.57860 |  0.12893 | 0.00790
-------
```

> Is this something we can achieve based on events? For example, aggregate diff between GET_BATCH_STARTED and GET_BATCH_COMPLETED?

Either like that or keep the previous way to measure. Pick the one you think the simplest.
I'm done with profiler implementation. The output looks as follows:
![image](https://user-images.githubusercontent.com/20420308/96625000-33e37c00-132b-11eb-83ae-ab5cc524d520.png)

You can review the [changes ](https://github.com/harsh8398/ignite/commit/8c987b6165a51dceb8db350580eebfe59bf87e8a)and suggest if any modifications required but there's still some TODOs left before raising the PR. Which are as follows:

- separate rows for Total, Dataflow and Processing stats
- type hints
- update `write_results()` as per new profiler stats format
- update tests

I have one question, should we include handler times for GET_BATCH_STARTED and GET_BATCH_COMPLETED for calculating dataflow time? Previous implementation had this but I'm not sure if it was intentional.
@harsh8398 table looks very cool 👍  Thanks a lot for working on that ! 
If you'd like to send a draft, please feel free to do that.

I'm just thinking about BC compatibility and maybe it would be better to keep 2 classes and name your version as `TimeProfiler` or `HandlersTimeProfiler` or another better name.

>  I have one question, should we include handler times for GET_BATCH_STARTED and GET_BATCH_COMPLETED for calculating dataflow time? Previous implementation had this but I'm not sure if it was intentional.

Probably, it would be cool ! And even (in another PR) if we could also handle customly added events and measure the time in attached handlers.


@vfdev-5 Okay then for BC it seems the best option is to add a new class as `HandlersTimeProfiler`. I'll do that.

For handling custom events, its a bit tricky because like **epoch and iteration event handlers** we don't know how many times they will get triggered. We can use something like this (https://discuss.pytorch.org/t/dynamically-extend-the-tensor/39553/2) for dynamically extending tensor whenever the handler is called. But I think it would get called so many times and it may consume time.
> @vfdev-5 Okay then for BC it seems the best option is to add a new class as HandlersTimeProfiler. I'll do that.

@harsh8398 sounds good !


> For handling custom events, its a bit tricky because like epoch and iteration event handlers we don't know how many times they will get triggered. We can use something like this (https://discuss.pytorch.org/t/dynamically-extend-the-tensor/39553/2) for dynamically extending tensor whenever the handler is called. But I think it would get called so many times and it may consume time.

Let's see this a bit later. I think handling custom events should not be too different from original events. Engine is aware of all events it could trigger and which handlers... 
If you could open a follow-up issue on that, it would be perfect :)


Resolving patch:
diff --git a/ignite/contrib/handlers/__init__.py b/ignite/contrib/handlers/__init__.py
--- a/ignite/contrib/handlers/__init__.py
+++ b/ignite/contrib/handlers/__init__.py
@@ -13,7 +13,7 @@
 from ignite.contrib.handlers.polyaxon_logger import PolyaxonLogger
 from ignite.contrib.handlers.stores import EpochOutputStore
 from ignite.contrib.handlers.tensorboard_logger import TensorboardLogger
-from ignite.contrib.handlers.time_profilers import BasicTimeProfiler
+from ignite.contrib.handlers.time_profilers import BasicTimeProfiler, HandlersTimeProfiler
 from ignite.contrib.handlers.tqdm_logger import ProgressBar
 from ignite.contrib.handlers.trains_logger import TrainsLogger
 from ignite.contrib.handlers.visdom_logger import VisdomLogger
diff --git a/ignite/contrib/handlers/time_profilers.py b/ignite/contrib/handlers/time_profilers.py
--- a/ignite/contrib/handlers/time_profilers.py
+++ b/ignite/contrib/handlers/time_profilers.py
@@ -1,9 +1,10 @@
+import functools
 from collections import OrderedDict
-from typing import Dict, Iterable, Mapping, Sequence, Union
+from typing import Any, Callable, Dict, Iterable, List, Mapping, Sequence, Union
 
 import torch
 
-from ignite.engine import Engine, Events
+from ignite.engine import Engine, EventEnum, Events
 from ignite.handlers import Timer
 
 
@@ -437,3 +438,324 @@ def odict_to_str(d: Mapping):
         )
         print(output_message)
         return output_message
+
+
+class HandlersTimeProfiler:
+    """
+    HandlersTimeProfiler can be used to profile the handlers,
+    data loading and data processing times. Custom events are also
+    profiled by this profiler
+
+    Examples:
+
+    .. code-block:: python
+
+        from ignite.contrib.handlers import HandlersTimeProfiler
+
+        trainer = Engine(train_updater)
+
+        # Create an object of the profiler and attach an engine to it
+        profiler = HandlersTimeProfiler()
+        profiler.attach(trainer)
+
+        @trainer.on(Events.EPOCH_COMPLETED)
+        def log_intermediate_results():
+            profiler.print_results(profiler.get_results())
+
+        trainer.run(dataloader, max_epochs=3)
+
+        profiler.write_results('path_to_dir/time_profiling.csv')
+
+    """
+
+    EVENT_FILTER_THESHOLD_TIME = 0.0001
+
+    def __init__(self):
+        self._dataflow_timer = Timer()
+        self._processing_timer = Timer()
+        self._event_handlers_timer = Timer()
+
+        self.dataflow_times = None
+        self.processing_times = None
+        self.event_handlers_times = None
+
+    @staticmethod
+    def _get_callable_name(handler: Callable) -> str:
+        # get name of the callable handler
+        return getattr(handler, "__qualname__", handler.__class__.__name__)
+
+    def _create_wrapped_handler(self, handler: Callable, event: Events) -> Callable:
+        @functools.wraps(handler)
+        def _timeit_handler(*args: Any, **kwargs: Any) -> None:
+            self._event_handlers_timer.reset()
+            handler(*args, **kwargs)
+            t = self._event_handlers_timer.value()
+            hname = self._get_callable_name(handler)
+            # filter profiled time if the handler was attached to event with event filter
+            if not hasattr(handler, "_parent") or t >= self.EVENT_FILTER_THESHOLD_TIME:
+                self.event_handlers_times[event][hname].append(t)
+
+        # required to revert back to original handler after profiling
+        setattr(_timeit_handler, "_profiler_original", handler)
+        return _timeit_handler
+
+    def _timeit_processing(self) -> None:
+        # handler used for profiling processing times
+        t = self._processing_timer.value()
+        self.processing_times.append(t)
+
+    def _timeit_dataflow(self) -> None:
+        # handler used for profiling dataflow times
+        t = self._dataflow_timer.value()
+        self.dataflow_times.append(t)
+
+    def _reset(self, event_handlers_names: Mapping[EventEnum, List[str]]) -> None:
+        # reset the variables used for profiling
+        self.dataflow_times = []
+        self.processing_times = []
+
+        self.event_handlers_times = {e: {h: [] for h in event_handlers_names[e]} for e in event_handlers_names}
+
+    @staticmethod
+    def _is_internal_handler(handler: Callable) -> bool:
+        # checks whether the handler is internal
+        return any(n in repr(handler) for n in ["HandlersTimeProfiler.", "Timer."])
+
+    def _detach_profiler_handlers(self, engine: Engine) -> None:
+        # reverts handlers to original handlers
+        for e in engine._event_handlers:
+            for i, (func, args, kwargs) in enumerate(engine._event_handlers[e]):
+                if hasattr(func, "_profiler_original"):
+                    engine._event_handlers[e][i] = (func._profiler_original, args, kwargs)
+
+    def _as_first_started(self, engine: Engine) -> None:
+        # wraps original handlers for profiling
+
+        self.event_handlers_names = {
+            e: [
+                self._get_callable_name(h)
+                for (h, _, _) in engine._event_handlers[e]
+                if not self._is_internal_handler(h)
+            ]
+            for e in engine._allowed_events
+        }
+
+        self._reset(self.event_handlers_names)
+
+        for e in engine._allowed_events:
+            for i, (func, args, kwargs) in enumerate(engine._event_handlers[e]):
+                if not self._is_internal_handler(func):
+                    engine._event_handlers[e][i] = (self._create_wrapped_handler(func, e), args, kwargs)
+
+        # processing timer
+        engine.add_event_handler(Events.ITERATION_STARTED, self._processing_timer.reset)
+        engine._event_handlers[Events.ITERATION_COMPLETED].insert(0, (self._timeit_processing, (), {}))
+
+        # dataflow timer
+        engine.add_event_handler(Events.GET_BATCH_STARTED, self._dataflow_timer.reset)
+        engine._event_handlers[Events.GET_BATCH_COMPLETED].insert(0, (self._timeit_dataflow, (), {}))
+
+        # revert back the wrapped handlers with original handlers at the end
+        engine.add_event_handler(Events.COMPLETED, self._detach_profiler_handlers)
+
+    def attach(self, engine: Engine) -> None:
+        if not isinstance(engine, Engine):
+            raise TypeError("Argument engine should be ignite.engine.Engine, but given {}".format(type(engine)))
+
+        if not engine.has_event_handler(self._as_first_started):
+            engine._event_handlers[Events.STARTED].insert(0, (self._as_first_started, (engine,), {}))
+
+    def get_results(self) -> List[List[Union[str, float]]]:
+        """
+        Method to fetch the aggregated profiler results after the engine is run
+
+        .. code-block:: python
+
+            results = profiler.get_results()
+
+        """
+        total_eh_time = sum(
+            [
+                sum(self.event_handlers_times[e][h])
+                for e in self.event_handlers_times
+                for h in self.event_handlers_times[e]
+            ]
+        )
+        total_eh_time = round(float(total_eh_time), 5,)
+
+        def compute_basic_stats(data: Sequence) -> List[Union[str, float]]:
+            data = torch.tensor(data, dtype=torch.float32)
+            # compute on non-zero data:
+            data = data[data > 0]
+            total = round(torch.sum(data).item(), 5) if len(data) > 0 else "not triggered"
+            min_index = ("None", "None")
+            max_index = ("None", "None")
+            mean = "None"
+            std = "None"
+            if len(data) > 0:
+                min_index = (round(torch.min(data).item(), 5), torch.argmin(data).item())
+                max_index = (round(torch.max(data).item(), 5), torch.argmax(data).item())
+                mean = round(torch.mean(data).item(), 5)
+                if len(data) > 1:
+                    std = round(torch.std(data).item(), 5)
+            return [total, min_index, max_index, mean, std]
+
+        event_handler_stats = [
+            [
+                h,
+                getattr(e, "name", str(e)),
+                *compute_basic_stats(torch.tensor(self.event_handlers_times[e][h], dtype=torch.float32)),
+            ]
+            for e in self.event_handlers_times
+            for h in self.event_handlers_times[e]
+        ]
+        event_handler_stats.append(["Total", "", total_eh_time, "", "", "", ""])
+        event_handler_stats.append(["Processing", "None", *compute_basic_stats(self.processing_times)])
+        event_handler_stats.append(["Dataflow", "None", *compute_basic_stats(self.dataflow_times)])
+
+        return event_handler_stats
+
+    def write_results(self, output_path: str) -> None:
+        """
+        Method to store the unaggregated profiling results to a csv file
+
+        .. code-block:: python
+
+            profiler.write_results('path_to_dir/awesome_filename.csv')
+
+        Example output:
+
+        .. code-block:: text
+
+            -----------------------------------------------------------------
+            # processing_stats dataflow_stats training.<locals>.log_elapsed_time (EPOCH_COMPLETED) ...
+            1     0.00003         0.252387                          0.125676
+            2     0.00029         0.252342                          0.125123
+
+        """
+        try:
+            import pandas as pd
+        except ImportError:
+            print("Need pandas to write results as files")
+            return
+
+        processing_stats = torch.tensor(self.processing_times, dtype=torch.float32)
+        dataflow_stats = torch.tensor(self.dataflow_times, dtype=torch.float32)
+
+        cols = [processing_stats, dataflow_stats]
+        headers = ["processing_stats", "dataflow_stats"]
+        for e in self.event_handlers_times:
+            for h in self.event_handlers_times[e]:
+                headers.append("{} ({})".format(h, getattr(e, "name", str(e))))
+                cols.append(torch.tensor(self.event_handlers_times[e][h], dtype=torch.float32))
+        # Determine maximum length
+        max_len = max([x.numel() for x in cols])
+
+        count_col = torch.arange(max_len, dtype=torch.float32) + 1
+        cols.insert(0, count_col)
+        headers.insert(0, "#")
+
+        # pad all tensors to have same length
+        cols = [torch.nn.functional.pad(x, pad=(0, max_len - x.numel()), mode="constant", value=0) for x in cols]
+
+        results_dump = torch.stack(cols, dim=1,).numpy()
+
+        results_df = pd.DataFrame(data=results_dump, columns=headers,)
+        results_df.to_csv(output_path, index=False)
+
+    @staticmethod
+    def print_results(results: List[List[Union[str, float]]]) -> None:
+        """
+        Method to print the aggregated results from the profiler
+
+        .. code-block:: python
+
+            profiler.print_results(results)
+
+        Example output:
+
+        .. code-block:: text
+
+            -----------------------------------------  -----------------------  -------------- ...
+            Handler                                    Event Name                     Total(s)
+            -----------------------------------------  -----------------------  --------------
+            run.<locals>.log_training_results          EPOCH_COMPLETED                19.43245
+            run.<locals>.log_validation_results        EPOCH_COMPLETED                 2.55271
+            run.<locals>.log_time                      EPOCH_COMPLETED                 0.00049
+            run.<locals>.log_intermediate_results      EPOCH_COMPLETED                 0.00106
+            run.<locals>.log_training_loss             ITERATION_COMPLETED               0.059
+            run.<locals>.log_time                      COMPLETED                 not triggered
+            -----------------------------------------  -----------------------  --------------
+            Total                                                                     22.04571
+            -----------------------------------------  -----------------------  --------------
+            Processing took total 11.29543s [min/index: 0.00393s/1875, max/index: 0.00784s/0,
+             mean: 0.00602s, std: 0.00034s]
+            Dataflow took total 16.24365s [min/index: 0.00533s/1874, max/index: 0.01129s/937,
+             mean: 0.00866s, std: 0.00113s]
+
+        """
+        # adopted implementation of torch.autograd.profiler.build_table
+        handler_column_width = max([len(item[0]) for item in results]) + 4
+        event_column_width = max([len(item[1]) for item in results]) + 4
+
+        DEFAULT_COLUMN_WIDTH = 14
+
+        headers = [
+            "Handler",
+            "Event Name",
+            "Total(s)",
+            "Min(s)/IDX",
+            "Max(s)/IDX",
+            "Mean(s)",
+            "Std(s)",
+        ]
+
+        # Have to use a list because nonlocal is Py3 only...
+        SPACING_SIZE = 2
+        row_format_lst = [""]
+        header_sep_lst = [""]
+        line_length_lst = [-SPACING_SIZE]
+
+        def add_column(padding: int, text_dir: str = ">") -> None:
+            row_format_lst[0] += "{: " + text_dir + str(padding) + "}" + (" " * SPACING_SIZE)
+            header_sep_lst[0] += "-" * padding + (" " * SPACING_SIZE)
+            line_length_lst[0] += padding + SPACING_SIZE
+
+        add_column(handler_column_width, text_dir="<")
+        add_column(event_column_width, text_dir="<")
+        for _ in headers[2:]:
+            add_column(DEFAULT_COLUMN_WIDTH)
+
+        row_format = row_format_lst[0]
+        header_sep = header_sep_lst[0]
+
+        result = []
+
+        def append(s: str) -> None:
+            result.append(s)
+            result.append("\n")
+
+        result.append("\n")
+        append(header_sep)
+        append(row_format.format(*headers))
+        append(header_sep)
+
+        for row in results[:-3]:
+            # format min/idx and max/idx
+            row[3] = "{}/{}".format(*row[3])
+            row[4] = "{}/{}".format(*row[4])
+
+            append(row_format.format(*row))
+
+        append(header_sep)
+        # print total handlers time row
+        append(row_format.format(*results[-3]))
+        append(header_sep)
+
+        summary_format = "{} took total {}s [min/index: {}, max/index: {}, mean: {}s, std: {}s]"
+        for row in results[-2:]:
+            row[3] = "{}s/{}".format(*row[3])
+            row[4] = "{}s/{}".format(*row[4])
+            del row[1]
+            append(summary_format.format(*row))
+        print("".join(result))


Test patch:
diff --git a/tests/ignite/contrib/handlers/test_time_profilers.py b/tests/ignite/contrib/handlers/test_time_profilers.py
--- a/tests/ignite/contrib/handlers/test_time_profilers.py
+++ b/tests/ignite/contrib/handlers/test_time_profilers.py
@@ -5,8 +5,8 @@
 import pytest
 from pytest import approx
 
-from ignite.contrib.handlers.time_profilers import BasicTimeProfiler
-from ignite.engine import Engine, Events
+from ignite.contrib.handlers.time_profilers import BasicTimeProfiler, HandlersTimeProfiler
+from ignite.engine import Engine, EventEnum, Events
 
 if sys.platform.startswith("darwin"):
     pytest.skip("Skip if on MacOS", allow_module_level=True)
@@ -16,7 +16,7 @@ def _do_nothing_update_fn(engine, batch):
     pass
 
 
-def get_prepared_engine(true_event_handler_time):
+def get_prepared_engine_for_basic_profiler(true_event_handler_time):
     dummy_trainer = Engine(_do_nothing_update_fn)
 
     @dummy_trainer.on(Events.STARTED)
@@ -54,7 +54,70 @@ def delay_get_batch_completed(engine):
     return dummy_trainer
 
 
-def test_dataflow_timer():
+def get_prepared_engine_for_handlers_profiler(true_event_handler_time):
+    HANDLERS_SLEEP_COUNT = 11
+    PROCESSING_SLEEP_COUNT = 3
+
+    class CustomEvents(EventEnum):
+        CUSTOM_STARTED = "custom_started"
+        CUSTOM_COMPLETED = "custom_completed"
+
+    def dummy_train_step(engine, batch):
+        engine.fire_event(CustomEvents.CUSTOM_STARTED)
+        time.sleep(true_event_handler_time)
+        engine.fire_event(CustomEvents.CUSTOM_COMPLETED)
+
+    dummy_trainer = Engine(dummy_train_step)
+    dummy_trainer.register_events(*CustomEvents)
+
+    @dummy_trainer.on(Events.STARTED)
+    def delay_start(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(Events.COMPLETED)
+    def delay_complete(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(Events.EPOCH_STARTED)
+    def delay_epoch_start(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(Events.EPOCH_COMPLETED)
+    def delay_epoch_complete(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(Events.ITERATION_STARTED)
+    def delay_iter_start(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(Events.ITERATION_COMPLETED)
+    def delay_iter_complete(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(Events.GET_BATCH_STARTED)
+    def delay_get_batch_started(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(Events.GET_BATCH_COMPLETED)
+    def delay_get_batch_completed(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(CustomEvents.CUSTOM_STARTED)
+    def delay_custom_started(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(CustomEvents.CUSTOM_COMPLETED)
+    def delay_custom_completed(engine):
+        time.sleep(true_event_handler_time)
+
+    @dummy_trainer.on(Events.EPOCH_STARTED(once=1))
+    def do_something_once_on_1_epoch():
+        time.sleep(true_event_handler_time)
+
+    return dummy_trainer, HANDLERS_SLEEP_COUNT, PROCESSING_SLEEP_COUNT
+
+
+def test_dataflow_timer_basic_profiler():
     true_dataflow_time_per_ele = 0.1
     true_max_epochs = 1
     true_num_iters = 2
@@ -81,7 +144,42 @@ def dummy_data_loader(data):
     assert dataflow_results["total"] == approx(true_num_iters * true_dataflow_time_per_ele, abs=1e-1)
 
 
-def test_processing_timer():
+def test_dataflow_timer_handlers_profiler():
+    true_dataflow_time_per_ele = 0.1
+    true_max_epochs = 1
+    true_num_iters = 2
+
+    def dummy_data_loader(data):
+        while True:
+            for d in data:
+                time.sleep(true_dataflow_time_per_ele)
+                yield d
+
+    dummy_data = range(true_num_iters)
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+    dummy_trainer.run(dummy_data_loader(dummy_data), max_epochs=true_max_epochs, epoch_length=true_num_iters)
+    results = profiler.get_results()
+    dataflow_results = results[-1]
+
+    assert dataflow_results[0] == "Dataflow"
+    # event name
+    assert dataflow_results[1] == "None"
+    # total
+    assert dataflow_results[2] == approx(true_num_iters * true_dataflow_time_per_ele, abs=1e-1)
+    # min
+    assert dataflow_results[3][0] == approx(true_dataflow_time_per_ele, abs=1e-1)
+    # max
+    assert dataflow_results[4][0] == approx(true_dataflow_time_per_ele, abs=1e-1)
+    # mean
+    assert dataflow_results[5] == approx(true_dataflow_time_per_ele, abs=1e-1)
+    # stddev
+    assert dataflow_results[6] == approx(0.0, abs=1e-1)
+
+
+def test_processing_timer_basic_profiler():
     true_processing_time = 0.1
     true_max_epochs = 2
     true_num_iters = 2
@@ -103,7 +201,37 @@ def train_updater(engine, batch):
     assert processing_results["total"] == approx(true_max_epochs * true_num_iters * true_processing_time, abs=1e-1)
 
 
-def test_event_handler_started():
+def test_processing_timer_handlers_profiler():
+    true_processing_time = 0.1
+    true_max_epochs = 2
+    true_num_iters = 2
+
+    def train_updater(engine, batch):
+        time.sleep(true_processing_time)
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(train_updater)
+    profiler.attach(dummy_trainer)
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    processing_results = results[-2]
+
+    assert processing_results[0] == "Processing"
+    # event name
+    assert processing_results[1] == "None"
+    # total
+    assert processing_results[2] == approx(true_max_epochs * true_num_iters * true_processing_time, abs=1e-1)
+    # min
+    assert processing_results[3][0] == approx(true_processing_time, abs=1e-1)
+    # max
+    assert processing_results[4][0] == approx(true_processing_time, abs=1e-1)
+    # mean
+    assert processing_results[5] == approx(true_processing_time, abs=1e-1)
+    # stddev
+    assert processing_results[6] == approx(0.0, abs=1e-1)
+
+
+def test_event_handler_started_basic_profiler():
     true_event_handler_time = 0.1
     true_max_epochs = 2
     true_num_iters = 2
@@ -123,7 +251,30 @@ def delay_start(engine):
     assert event_results["total"] == approx(true_event_handler_time, abs=1e-1)
 
 
-def test_event_handler_completed():
+def test_event_handler_started_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 2
+    true_num_iters = 2
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.STARTED)
+    def delay_start(engine):
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+
+    assert "delay_start" in event_results[0]
+    assert event_results[1] == "STARTED"
+
+    assert event_results[2] == approx(true_event_handler_time, abs=1e-1)  # total
+
+
+def test_event_handler_completed_basic_profiler():
     true_event_handler_time = 0.1
     true_max_epochs = 2
     true_num_iters = 2
@@ -143,7 +294,29 @@ def delay_complete(engine):
     assert event_results["total"] == approx(true_event_handler_time, abs=1e-1)
 
 
-def test_event_handler_epoch_started():
+def test_event_handler_completed_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 2
+    true_num_iters = 2
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.COMPLETED)
+    def delay_complete(engine):
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+    assert "delay_complete" in event_results[0]
+    assert event_results[1] == "COMPLETED"
+
+    assert event_results[2] == approx(true_event_handler_time, abs=1e-1)  # total
+
+
+def test_event_handler_epoch_started_basic_profiler():
     true_event_handler_time = 0.1
     true_max_epochs = 2
     true_num_iters = 1
@@ -167,7 +340,33 @@ def delay_epoch_start(engine):
     assert event_results["total"] == approx(true_max_epochs * true_event_handler_time, abs=1e-1)
 
 
-def test_event_handler_epoch_completed():
+def test_event_handler_epoch_started_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 2
+    true_num_iters = 1
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.EPOCH_STARTED)
+    def delay_epoch_start(engine):
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+    assert "delay_epoch_start" in event_results[0]
+    assert event_results[1] == "EPOCH_STARTED"
+
+    assert event_results[2] == approx(true_max_epochs * true_event_handler_time, abs=1e-1)  # total
+    assert event_results[3][0] == approx(true_event_handler_time, abs=1e-1)  # min
+    assert event_results[4][0] == approx(true_event_handler_time, abs=1e-1)  # max
+    assert event_results[5] == approx(true_event_handler_time, abs=1e-1)  # mean
+    assert event_results[6] == approx(0.0, abs=1e-1)  # stddev
+
+
+def test_event_handler_epoch_completed_basic_profiler():
     true_event_handler_time = 0.1
     true_max_epochs = 2
     true_num_iters = 1
@@ -191,7 +390,33 @@ def delay_epoch_complete(engine):
     assert event_results["total"] == approx(true_max_epochs * true_event_handler_time, abs=1e-1)
 
 
-def test_event_handler_iteration_started():
+def test_event_handler_epoch_completed_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 2
+    true_num_iters = 1
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.EPOCH_COMPLETED)
+    def delay_epoch_complete(engine):
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+    assert "delay_epoch_complete" in event_results[0]
+    assert event_results[1] == "EPOCH_COMPLETED"
+
+    assert event_results[2] == approx(true_max_epochs * true_event_handler_time, abs=1e-1)  # total
+    assert event_results[3][0] == approx(true_event_handler_time, abs=1e-1)  # min
+    assert event_results[4][0] == approx(true_event_handler_time, abs=1e-1)  # max
+    assert event_results[5] == approx(true_event_handler_time, abs=1e-1)  # mean
+    assert event_results[6] == approx(0.0, abs=1e-1)  # stddev
+
+
+def test_event_handler_iteration_started_basic_profiler():
     true_event_handler_time = 0.1
     true_max_epochs = 1
     true_num_iters = 2
@@ -215,7 +440,33 @@ def delay_iter_start(engine):
     assert event_results["total"] == approx(true_max_epochs * true_num_iters * true_event_handler_time, abs=1e-1)
 
 
-def test_event_handler_iteration_completed():
+def test_event_handler_iteration_started_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 1
+    true_num_iters = 2
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.ITERATION_STARTED)
+    def delay_iter_start(engine):
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+    assert "delay_iter_start" in event_results[0]
+    assert event_results[1] == "ITERATION_STARTED"
+
+    assert event_results[2] == approx(true_max_epochs * true_num_iters * true_event_handler_time, abs=1e-1)  # total
+    assert event_results[3][0] == approx(true_event_handler_time, abs=1e-1)  # min
+    assert event_results[4][0] == approx(true_event_handler_time, abs=1e-1)  # max
+    assert event_results[5] == approx(true_event_handler_time, abs=1e-1)  # mean
+    assert event_results[6] == approx(0.0, abs=1e-1)  # stddev
+
+
+def test_event_handler_iteration_completed_basic_profiler():
     true_event_handler_time = 0.1
     true_max_epochs = 1
     true_num_iters = 2
@@ -239,7 +490,33 @@ def delay_iter_complete(engine):
     assert event_results["total"] == approx(true_max_epochs * true_num_iters * true_event_handler_time, abs=1e-1)
 
 
-def test_event_handler_get_batch_started():
+def test_event_handler_iteration_completed_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 1
+    true_num_iters = 2
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.ITERATION_COMPLETED)
+    def delay_iter_complete(engine):
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+    assert "delay_iter_complete" in event_results[0]
+    assert event_results[1] == "ITERATION_COMPLETED"
+
+    assert event_results[2] == approx(true_max_epochs * true_num_iters * true_event_handler_time, abs=1e-1)  # total
+    assert event_results[3][0] == approx(true_event_handler_time, abs=1e-1)  # min
+    assert event_results[4][0] == approx(true_event_handler_time, abs=1e-1)  # max
+    assert event_results[5] == approx(true_event_handler_time, abs=1e-1)  # mean
+    assert event_results[6] == approx(0.0, abs=1e-1)  # stddev
+
+
+def test_event_handler_get_batch_started_basic_profiler():
     true_event_handler_time = 0.1
     true_max_epochs = 1
     true_num_iters = 2
@@ -263,6 +540,32 @@ def delay_get_batch_started(engine):
     assert event_results["total"] == approx(true_max_epochs * true_num_iters * true_event_handler_time, abs=1e-1)
 
 
+def test_event_handler_get_batch_started_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 1
+    true_num_iters = 2
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.GET_BATCH_STARTED)
+    def delay_get_batch_started(engine):
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+    assert "delay_get_batch_started" in event_results[0]
+    assert event_results[1] == "GET_BATCH_STARTED"
+
+    assert event_results[2] == approx(true_max_epochs * true_num_iters * true_event_handler_time, abs=1e-1)  # total
+    assert event_results[3][0] == approx(true_event_handler_time, abs=1e-1)  # min
+    assert event_results[4][0] == approx(true_event_handler_time, abs=1e-1)  # max
+    assert event_results[5] == approx(true_event_handler_time, abs=1e-1)  # mean
+    assert event_results[6] == approx(0.0, abs=1e-1)  # stddev
+
+
 def test_event_handler_get_batch_completed():
     true_event_handler_time = 0.1
     true_max_epochs = 1
@@ -287,7 +590,114 @@ def delay_get_batch_completed(engine):
     assert event_results["total"] == approx(true_max_epochs * true_num_iters * true_event_handler_time, abs=1e-1)
 
 
-def test_event_handler_total_time():
+def test_event_handler_get_batch_completed_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 1
+    true_num_iters = 2
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.GET_BATCH_COMPLETED)
+    def delay_get_batch_completed(engine):
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+    assert "delay_get_batch_completed" in event_results[0]
+    assert event_results[1] == "GET_BATCH_COMPLETED"
+
+    assert event_results[2] == approx(true_max_epochs * true_num_iters * true_event_handler_time, abs=1e-1)  # total
+    assert event_results[3][0] == approx(true_event_handler_time, abs=1e-1)  # min
+    assert event_results[4][0] == approx(true_event_handler_time, abs=1e-1)  # max
+    assert event_results[5] == approx(true_event_handler_time, abs=1e-1)  # mean
+    assert event_results[6] == approx(0.0, abs=1e-1)  # stddev
+
+
+def test_neg_event_filter_threshold_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 1
+    true_num_iters = 1
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.EPOCH_STARTED(once=2))
+    def do_something_once_on_2_epoch():
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+    assert "do_something_once_on_2_epoch" in event_results[0]
+    assert event_results[1] == "EPOCH_STARTED"
+    assert event_results[2] == "not triggered"
+
+
+def test_pos_event_filter_threshold_handlers_profiler():
+    true_event_handler_time = HandlersTimeProfiler.EVENT_FILTER_THESHOLD_TIME
+    true_max_epochs = 2
+    true_num_iters = 1
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.EPOCH_STARTED(once=2))
+    def do_something_once_on_2_epoch():
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = results[0]
+    assert "do_something_once_on_2_epoch" in event_results[0]
+    assert event_results[1] == "EPOCH_STARTED"
+    assert event_results[2] == approx(
+        (true_max_epochs * true_num_iters * true_event_handler_time) / 2, abs=1e-1
+    )  # total
+
+
+def test_custom_event_with_arg_handlers_profiler():
+    true_event_handler_time = 0.1
+    true_max_epochs = 1
+    true_num_iters = 2
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer = Engine(_do_nothing_update_fn)
+    dummy_trainer.register_events("custom_event")
+    profiler.attach(dummy_trainer)
+
+    @dummy_trainer.on(Events.ITERATION_COMPLETED(every=1))
+    def trigger_custom_event():
+        dummy_trainer.fire_event("custom_event")
+
+    args = [122, 324]
+
+    @dummy_trainer.on("custom_event", args)
+    def on_custom_event(args):
+        time.sleep(true_event_handler_time)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    event_results = None
+    for row in results:
+        if row[1] == "custom_event":
+            event_results = row
+            break
+    assert event_results is not None
+    assert "on_custom_event" in event_results[0]
+
+    assert event_results[2] == approx(true_max_epochs * true_num_iters * true_event_handler_time, abs=1e-1)  # total
+    assert event_results[3][0] == approx(true_event_handler_time, abs=1e-1)  # min
+    assert event_results[4][0] == approx(true_event_handler_time, abs=1e-1)  # max
+    assert event_results[5] == approx(true_event_handler_time, abs=1e-1)  # mean
+    assert event_results[6] == approx(0.0, abs=1e-1)  # stddev
+
+
+def test_event_handler_total_time_basic_profiler():
     true_event_handler_time = 0.125
     true_max_epochs = 1
     true_num_iters = 1
@@ -335,13 +745,33 @@ def delay_get_batch_completed(engine):
     assert event_results["total_time"].item() == approx(true_event_handler_time * 8, abs=1e-1)
 
 
-def test_write_results(dirname):
+def test_event_handler_total_time_handlers_profiler():
+    true_event_handler_time = 0.125
+    true_max_epochs = 1
+    true_num_iters = 1
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer, handlers_sleep_count, processing_sleep_count = get_prepared_engine_for_handlers_profiler(
+        true_event_handler_time
+    )
+    profiler.attach(dummy_trainer)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    results = profiler.get_results()
+    total_handler_stats = results[-3]  # total result row
+    total_processing_stats = results[-2]  # processing result row
+
+    assert total_handler_stats[2] == approx(true_event_handler_time * handlers_sleep_count, abs=1e-1)  # total time
+    assert total_processing_stats[2] == approx(true_event_handler_time * processing_sleep_count, abs=1e-1)  # total time
+
+
+def test_write_results_basic_profiler(dirname):
     true_event_handler_time = 0.125
     true_max_epochs = 3
     true_num_iters = 2
 
     profiler = BasicTimeProfiler()
-    dummy_trainer = get_prepared_engine(true_event_handler_time)
+    dummy_trainer = get_prepared_engine_for_basic_profiler(true_event_handler_time)
     profiler.attach(dummy_trainer)
 
     dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
@@ -358,13 +788,36 @@ def test_write_results(dirname):
     assert file_length == (true_max_epochs * true_num_iters) + 1
 
 
-def test_print_results(capsys):
+def test_write_results_handlers_profiler(dirname):
+    true_event_handler_time = 0.125
+    true_max_epochs = 3
+    true_num_iters = 2
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer, _, _ = get_prepared_engine_for_handlers_profiler(true_event_handler_time)
+    profiler.attach(dummy_trainer)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    fp = os.path.join(dirname, "test_log.csv")
+    profiler.write_results(fp)
+
+    assert os.path.isfile(fp)
+
+    file_length = 0
+    with open(fp) as f:
+        for l in f:
+            file_length += 1
+
+    assert file_length == (true_max_epochs * true_num_iters) + 1
+
+
+def test_print_results_basic_profiler(capsys):
 
     true_max_epochs = 1
     true_num_iters = 5
 
     profiler = BasicTimeProfiler()
-    dummy_trainer = get_prepared_engine(true_event_handler_time=0.0125)
+    dummy_trainer = get_prepared_engine_for_basic_profiler(true_event_handler_time=0.0125)
     profiler.attach(dummy_trainer)
 
     dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
@@ -376,13 +829,31 @@ def test_print_results(capsys):
     assert "nan" not in out
 
 
-def test_get_intermediate_results_during_run(capsys):
+def test_print_results_handlers_profiler_handlers_profiler(capsys):
+
+    true_max_epochs = 1
+    true_num_iters = 5
+
+    profiler = HandlersTimeProfiler()
+    dummy_trainer, _, _ = get_prepared_engine_for_handlers_profiler(true_event_handler_time=0.0125)
+    profiler.attach(dummy_trainer)
+
+    dummy_trainer.run(range(true_num_iters), max_epochs=true_max_epochs)
+    HandlersTimeProfiler.print_results(profiler.get_results())
+
+    captured = capsys.readouterr()
+    out = captured.out
+    assert "HandlersTimeProfiler." not in out
+    assert "Timer." not in out
+
+
+def test_get_intermediate_results_during_run_basic_profiler(capsys):
     true_event_handler_time = 0.0645
     true_max_epochs = 2
     true_num_iters = 5
 
     profiler = BasicTimeProfiler()
-    dummy_trainer = get_prepared_engine(true_event_handler_time)
+    dummy_trainer = get_prepared_engine_for_basic_profiler(true_event_handler_time)
     profiler.attach(dummy_trainer)
 
     @dummy_trainer.on(Events.ITERATION_COMPLETED(every=3))