Repository: pytorch/ignite

Issue:
gradient_accumulation_steps influences scale of the loss
## 🐛 Bug description

[Because of the division here](https://github.com/pytorch/ignite/blob/master/ignite/engine/__init__.py#L185), and similar functions, the loss has the wrong scale when using `gradient_accumulation_steps`. This makes it confusingly low in comparison to the valid loss.

One option is to only divide on the backward call, i.e. doing this:
```
scaler.scale(loss / gradient_accumulation_steps).backward()
```
or one could multiply the loss again with `gradient_accumulation_steps` before returning it.


Discussion:
@RuABraun thanks for reporting ! 

If i get your point correctly, the fact that loss is scaled by `gradient_accumulation_steps` and this value is assigned to the output loss, it is confusing that values are not comparable to ones without `gradient_accumulation_steps` ?
I agree that division is needed for gradients computation.

I think multiplying the output loss as `return output_transform(x, y, y_pred, loss * gradient_accumulation_steps)` could be good enough

I'd like to work on this one

Resolving patch:
diff --git a/ignite/engine/__init__.py b/ignite/engine/__init__.py
--- a/ignite/engine/__init__.py
+++ b/ignite/engine/__init__.py
@@ -106,7 +106,7 @@ def update(engine: Engine, batch: Sequence[torch.Tensor]) -> Union[Any, Tuple[to
         loss.backward()
         if engine.state.iteration % gradient_accumulation_steps == 0:
             optimizer.step()
-        return output_transform(x, y, y_pred, loss)
+        return output_transform(x, y, y_pred, loss * gradient_accumulation_steps)
 
     return update
 
@@ -192,7 +192,7 @@ def update(engine: Engine, batch: Sequence[torch.Tensor]) -> Union[Any, Tuple[to
             loss.backward()
             if engine.state.iteration % gradient_accumulation_steps == 0:
                 optimizer.step()
-        return output_transform(x, y, y_pred, loss)
+        return output_transform(x, y, y_pred, loss * gradient_accumulation_steps)
 
     return update
 
@@ -269,7 +269,7 @@ def update(engine: Engine, batch: Sequence[torch.Tensor]) -> Union[Any, Tuple[to
             scaled_loss.backward()
         if engine.state.iteration % gradient_accumulation_steps == 0:
             optimizer.step()
-        return output_transform(x, y, y_pred, loss)
+        return output_transform(x, y, y_pred, loss * gradient_accumulation_steps)
 
     return update
 
@@ -344,7 +344,7 @@ def update(engine: Engine, batch: Sequence[torch.Tensor]) -> Union[Any, Tuple[to
         loss.backward()
         if engine.state.iteration % gradient_accumulation_steps == 0:
             xm.optimizer_step(optimizer, barrier=True)
-        return output_transform(x, y, y_pred, loss)
+        return output_transform(x, y, y_pred, loss * gradient_accumulation_steps)
 
     return update
 


Test patch:
diff --git a/tests/ignite/engine/test_create_supervised.py b/tests/ignite/engine/test_create_supervised.py
--- a/tests/ignite/engine/test_create_supervised.py
+++ b/tests/ignite/engine/test_create_supervised.py
@@ -97,8 +97,8 @@ def _():
         _x, _y = trainer.state.batch
         _x, _y = _x.to(model_device), _y.to(model_device)
         accumulation[0] += 0.2 * _x.item() * (theta[0] * _x.item() - _y.item())
-        # loss is not accumulated !
-        loss[0] = mse_loss(model(_x), _y).item() / gradient_accumulation_steps
+        # value of loss should not be accumulated
+        loss[0] = mse_loss(model(_x), _y).item()
 
     @trainer.on(Events.ITERATION_COMPLETED(every=gradient_accumulation_steps))
     def _():