Repository: pytorch/ignite

Issue:
Feature Request: add an additional argument to auto_optim to allow for gradient accumulation
When we use horovod backend and perform gradient accumulation, we get the following error:
`AssertionError: Gradients were computed more than backward_passes_per_step times before call to step(). Increase backward_passes_per_step to accumulate gradients
locally.`

Thus, we need to change the default argument `backward_passes_per_step` of `horovod.DistributedOptimizer` to enable gradient accumulation in the distributed setting.  To do so, we can add this argument to `ignite.distributed.auto_optim`.




Discussion:
@sandylaker thanks for the feature request ! Maybe, we can enable kwargs for `auto_optim` as it is done for `auto_model`. In the docs we can explicitly say where kwargs goes exactly.
Hey @vfdev-5 .
I would like to work on this issue.
Before I start I just want to know whether I have to add backward_passes_per_step as an argument into the ignite.distributed.auto_optim function?

Resolving patch:
diff --git a/ignite/distributed/auto.py b/ignite/distributed/auto.py
--- a/ignite/distributed/auto.py
+++ b/ignite/distributed/auto.py
@@ -234,7 +234,7 @@ def auto_model(model: nn.Module, sync_bn: bool = False, **kwargs: Any) -> nn.Mod
     return model
 
 
-def auto_optim(optimizer: Optimizer) -> Optimizer:
+def auto_optim(optimizer: Optimizer, **kwargs: Any) -> Optimizer:
     """Helper method to adapt optimizer for non-distributed and distributed configurations (supporting
     all available backends from :meth:`~ignite.distributed.utils.available_backends()`).
 
@@ -256,6 +256,7 @@ def auto_optim(optimizer: Optimizer) -> Optimizer:
 
     Args:
         optimizer: input torch optimizer
+        kwargs: kwargs to Horovod backend's DistributedOptimizer.
 
     Returns:
         Optimizer
@@ -264,6 +265,9 @@ def auto_optim(optimizer: Optimizer) -> Optimizer:
 
     .. versionchanged:: 0.4.2
         Added Horovod distributed optimizer.
+
+    .. versionchanged:: 0.5.0
+        Added kwargs to ``idist.auto_optim``.
     """
     bnd = idist.backend()
     if idist.has_xla_support and bnd == idist_xla.XLA_TPU:
@@ -273,7 +277,7 @@ def auto_optim(optimizer: Optimizer) -> Optimizer:
     if idist.has_hvd_support and bnd == idist_hvd.HOROVOD:
         import horovod.torch as hvd
 
-        optimizer = hvd.DistributedOptimizer(optimizer)
+        optimizer = hvd.DistributedOptimizer(optimizer, **kwargs)
         hvd.broadcast_optimizer_state(optimizer, root_rank=0)
         return optimizer
 


Test patch:
diff --git a/tests/ignite/distributed/test_auto.py b/tests/ignite/distributed/test_auto.py
--- a/tests/ignite/distributed/test_auto.py
+++ b/tests/ignite/distributed/test_auto.py
@@ -172,6 +172,13 @@ def _test_auto_model_optimizer(ws, device):
     else:
         assert isinstance(optimizer, optim.SGD) and not hasattr(optimizer, "wrapped_optimizer")
 
+    if idist.has_hvd_support and bnd in ("horovod",):
+        backward_passes_per_step = 2
+        optimizer = optim.SGD(model.parameters(), lr=0.01)
+        optimizer = auto_optim(optimizer, backward_passes_per_step=backward_passes_per_step)
+        assert isinstance(optimizer, optim.SGD) and hasattr(optimizer, "backward_passes_per_step")
+        assert optimizer.backward_passes_per_step == backward_passes_per_step
+
 
 def test_auto_methods_no_dist():