Repository: huggingface/accelerate

Issue:
Question/Bug about `accelerator.reduce` on single GPU
### System Info

```Shell
- `Accelerate` version: 0.18.0
- Platform: Linux-4.15.0-45-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.13
- Numpy version: 1.21.5
- PyTorch version (GPU?): 1.13.0 (True)
- `Accelerate` default config:
        Not found
```


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [X] My own task or dataset (give details below)

### Reproduction

Hi, there.

I am confused about the behavior of `accelerator.reduce` when running on a single GPU. As far as I know, it is used to sum up (or average) the tensors elementwisely across processes, thus is expected to do nothing if the code is running on a single process. However, this is not the case as shown below:

```python
import torch
from accelerate import Accelerator

accelerator = Accelerator()

tensor = torch.arange(4) + (4 * accelerator.process_index)
tensor = tensor.float().to(accelerator.device)

print(f'[Rank{accelerator.process_index}] before reduction: {tensor}')
# if single process:
#    Rank 0: [0., 1., 2., 3.]
# if two processes:
#    Rank 0: [0., 1., 2., 3.]
#    Rank 1: [4., 5., 6., 7.]

sum_reduced_tensor = accelerator.reduce(tensor, reduction="sum")
print(f'[Rank{accelerator.process_index}] sum reduction: {sum_reduced_tensor}')
# if single process:
#    Rank 0: 6.0
# if two processes:
#    Rank 0: [4., 6., 8., 10.]
#    Rank 1: [4., 6., 8., 10.]

mean_reduced_tensor = accelerator.reduce(tensor, reduction="mean")
print(f'[Rank{accelerator.process_index}] mean reduction: {mean_reduced_tensor}')
# if single process:
#    Rank 0: 1.5
# if two processes:
#    Rank 0: [2., 3., 4., 5.]
#    Rank 1: [2., 3., 4., 5.]
```

If the code is launched on two processes, everything's fine. The tensor is indeed reduced elementwisely.
However, if it is launched on a single process, the tensor is summed/averaged to a single value, but I think it should be kept unchanged.

I found the related source code [here](https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/operations.py#L436-L440):

```python
        if state.distributed_type == DistributedType.NO:
            if reduction == "sum":
                return cloned_tensor.sum()
            else:
                return cloned_tensor.mean()
```

And I think it should directly `return cloned_tensor` regardless of what `reduction` is.

Actually there's another issue related to this problem: https://github.com/huggingface/accelerate/issues/977, but it is closed due to lack of activity.

### Expected behavior

```Shell
accelerator.reduce should not change the tensor when running on single process.
```



Discussion:
Thanks for flagging! Since you identified the right fix, would you like to make a PR with it?
> Thanks for flagging! Since you identified the right fix, would you like to make a PR with it?

Sure. I'll make a PR later.

Resolving patch:
diff --git a/src/accelerate/utils/operations.py b/src/accelerate/utils/operations.py
--- a/src/accelerate/utils/operations.py
+++ b/src/accelerate/utils/operations.py
@@ -434,10 +434,7 @@ def _reduce_across_processes(tensor, reduction="mean"):
         state = PartialState()
         cloned_tensor = tensor.clone()
         if state.distributed_type == DistributedType.NO:
-            if reduction == "sum":
-                return cloned_tensor.sum()
-            else:
-                return cloned_tensor.mean()
+            return cloned_tensor
         if state.distributed_type == DistributedType.TPU:
             xm.all_reduce("sum", cloned_tensor)
         elif state.distributed_type.value in CUDA_DISTRIBUTED_TYPES:


Test patch: