diff --git a/src/accelerate/utils/operations.py b/src/accelerate/utils/operations.py
index 42868a0a5..3ba46a03f 100644
--- a/src/accelerate/utils/operations.py
+++ b/src/accelerate/utils/operations.py
@@ -103,7 +103,7 @@ def recursively_apply(func, data, *args, test_type=is_torch_tensor, error_on_oth
     return data
 
 
-def send_to_device(tensor, device):
+def send_to_device(tensor, device, non_blocking=False):
     """
     Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device.
 
@@ -117,13 +117,16 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        try:
+            return t.to(device, non_blocking=non_blocking)
+        except TypeError:  # .to() doesn't accept non_blocking as kwarg
+            return t.to(device)
 
     def _has_to_method(t):
         return hasattr(t, "to")
 
-    return recursively_apply(_send_to_device, tensor, device, test_type=_has_to_method)
+    return recursively_apply(_send_to_device, tensor, device, non_blocking, test_type=_has_to_method)
 
 
 def get_data_structure(data):


Review comments: body: So far all objects in Pytorch which accept `.to()` method accept the `non_blocking` kwarg or at least kwargs in general, but are we sure this is going to be always the case?

- [torch.jit.ScriptModule.to](https://pytorch.org/docs/stable/generated/torch.jit.ScriptModule.html?highlight=#torch.jit.ScriptModule.to) (Python method, in ScriptModule)
- [torch.nn.Module.to](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=#torch.nn.Module.to) (Python method, in Module)
- [torch.nn.utils.rnn.PackedSequence.to](https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.PackedSequence.html?highlight=#torch.nn.utils.rnn.PackedSequence.to) (Python method, in PackedSequence)
- [torch.Tensor.to](https://pytorch.org/docs/stable/generated/torch.Tensor.to.html?highlight=#torch.Tensor.to) (Python method, in torch.Tensor.to)

cc @sgugger 
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: This is for our own objects FYI, `BatchEncoding` in Transformers, which does not accept the `non_blocking` argument :-/
Maybe we can inspect the signature and only pass it when we see it?
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: I tried using `t.to.__code__` and `inspect.signature(t.to)` and both give errors. Do you have a better way to get kwargs of a `builtin_function_or_method` @sgugger ?
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: Mmm, `inspect.signature(t.to)` should give you access to the keyword arguments. You can find them in the `.parameters` attribute, which will be a dict-like object.
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: I get this error when trying `inspect.signature(t.to)` where `t` is a tensor
```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/nouamane/miniconda3/envs/hf/lib/python3.9/inspect.py", line 3111, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
  File "/home/nouamane/miniconda3/envs/hf/lib/python3.9/inspect.py", line 2860, in from_callable
    return _signature_from_callable(obj, sigcls=cls,
  File "/home/nouamane/miniconda3/envs/hf/lib/python3.9/inspect.py", line 2327, in _signature_from_callable
    return _signature_from_builtin(sigcls, obj,
  File "/home/nouamane/miniconda3/envs/hf/lib/python3.9/inspect.py", line 2145, in _signature_from_builtin
    raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in method to of Tensor object at 0x7f7e98c57720>
```
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: I think this is related to the warning in [inspect's docs](https://docs.python.org/3/library/inspect.html#inspect.signature)

> Note: Some callables may not be introspectable in certain implementations of Python. For example, in CPython, some built-in functions defined in C provide no metadata about their arguments.
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: Maybe we can isolate tensors in the function then? Something like:
```
if isinstance(t, torch.Tensor):
    return t.to(device, non_blocking=non_blocking)
else:
    # inspect signature here in a try/except block
```
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: Wouldn't this be enough, or should I inspect signature?
  ```python
  def _send_to_device(t, device, non_blocking):
        if isinstance(t, torch.Tensor):
            return t.to(device, non_blocking=non_blocking)
        else:
            try:
                return t.to(device, non_blocking=non_blocking)
            except TypeError:
                return t.to(device)
```
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: Oh in this case, you don't even need the first test. Are you sure about the type of `TypeError`?
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: Searching on google for `python unexpected argument`,  it's usually `TypeError: ... got an unexpected keyword`. I'm not sure if it's always the case though
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: Seems to work for C defined methods as well:
```python
>> t.__abs__(a=5)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: Tensor.abs() takes no keyword arguments
```


```python
>> t.to(a=5)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: to() received an invalid combination of arguments - got (a=int, ), but expected one of:
 * (torch.device device, torch.dtype dtype, bool non_blocking, bool copy, *, torch.memory_format memory_format)
 * (torch.dtype dtype, bool non_blocking, bool copy, *, torch.memory_format memory_format)
 * (Tensor tensor, bool non_blocking, bool copy, *, torch.memory_format memory_format)
```
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: Nice!
diff_hunk: @@ -117,13 +117,13 @@ def send_to_device(tensor, device):
         The same data structure as `tensor` with all tensors sent to the proper device.
     """
 
-    def _send_to_device(t, device):
-        return t.to(device)
+    def _send_to_device(t, device, non_blocking):
+        return t.to(device, non_blocking=non_blocking)
 
     def _has_to_method(t):
         return hasattr(t, "to")
body: @sgugger the docs say that `tensor` is a torch.Tensor here, which isn't necessary the case iiuc. We can probably update the docs as well
diff_hunk: @@ -103,7 +103,7 @@ def recursively_apply(func, data, *args, test_type=is_torch_tensor, error_on_oth
     return data
 
 
-def send_to_device(tensor, device):
+def send_to_device(tensor, device, non_blocking=False):
     """
     Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device.
body: True
diff_hunk: @@ -103,7 +103,7 @@ def recursively_apply(func, data, *args, test_type=is_torch_tensor, error_on_oth
     return data
 
 
-def send_to_device(tensor, device):
+def send_to_device(tensor, device, non_blocking=False):
     """
     Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device.
body: We could leave that for another PR 
diff_hunk: @@ -103,7 +103,7 @@ def recursively_apply(func, data, *args, test_type=is_torch_tensor, error_on_oth
     return data
 
 
-def send_to_device(tensor, device):
+def send_to_device(tensor, device, non_blocking=False):
     """
     Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device.

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/accelerate/blob/main/CONTRIBUTING.md) are likely to be ignored.
body: Hi @sgugger, was there a reason that this was closed and not merged? I came across this when I was about to request the same thing. 
body: @Chris-hughes10 it was automatically closed by the stale bot due to the absence of activity. @NouamaneTazi is still investigating whether this could have some negative impact sometimes.
body: ### Should we make .to(non_blocking=True) the default in **accelerate**?
Let’s use this script to find out

```python
import torch
if __name__ == '__main__':
    seed = 0
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    stream = torch.cuda.current_stream()

    x = torch.rand(32, 256, 220, 220).cuda()

    t = (x.min() - x.max()).to(torch.device("cpu"), non_blocking=False) # try with False then True
    # t = (x.min() - x.max()).to("cuda:1", non_blocking=True) # try GPU0 to GPU1 copy

    print(stream.query()) # False - Checks if all the work submitted has been completed.
    print(t)
    stream.synchronize() # wait for stream to finish the work
    
    print(stream.query()) # True - work done
    print(t)
```

<details>
<summary>Copy to CPU with non_blocking=False   (default)</summary>

In this case `.to()`  adds a `cudaStreamSynchronize` op which makes the CPU use the correct value of the tensor when printing
![image](https://user-images.githubusercontent.com/29777165/194038267-91033b35-fc18-4a0d-9af1-cfb1e382721e.png)

</details>

<details>
<summary>Copy to CPU with non_blocking=True  </summary>


In this case the CPU submits the kernels for `.to()`  to the GPU then moves on to perform the print operation which uses an incorrect value for the tensor `tensor(0.)`  **(The dangerous part)** 
![image](https://user-images.githubusercontent.com/29777165/194038545-88833ffc-5141-4fd4-a503-c140202f0bed.png)

</details>

<details>
<summary>Copy to another GPU with non_blocking=True </summary>

It seems that the `non_blocking` here doesn’t do much (we get basically the same thing using `non_blocking=True` ). In both cases we have GPU 1 waiting for GPU 0 to finish working on the tensor, and THEN copy it to GPU 1. And finally the CPU prints the tensor that’s now located on GPU 1
In this case `.to()`  creates a `cudaStreamWaitEvent` event (figure 2) which makes GPU 1 waits for GPU 0. I made [an issue](https://discuss.pytorch.org/t/to-cuda-1-non-blocking-true-creates-cudastreamwaitevent/162296) on Pytorch’s forums to investigate why is this the case

![image](https://user-images.githubusercontent.com/29777165/194038566-56a7c178-d659-4c3f-a759-d94da8c0f152.png)

![image](https://user-images.githubusercontent.com/29777165/194038587-5a089b5c-af42-4bab-86b1-516e2ebe3300.png)

</details>

**tldr;** `non_blocking` could be a game changer in using your GPUs efficiently. 
- **Good use scenario:** Copying your data from CPU to GPU in a non_blocking way then running your model which exists on GPU (this would make the CPU launch the copy kernel, then moves on to queuing other kernels in your model on the GPU. As opposed to waiting for the copy to end, and only then launching kernels from the model). [Example from Pytorch's repo](https://github.com/pytorch/examples/blob/main/imagenet/main.py#L321-L326).
- **Bad use scenario:** Copying your data from CPU to GPU in a non_blocking way then start some operations **on CPU** that would use the non-ready tensors. (could be in `if` statements, or simple arithmetics...)


=> It’s good to support that argument in accelerate but it’s better to keep the default as it is, just like it’s the case in Pytorch 
body: @NouamaneTazi These are really interesting insights! If you are looking at inspecting signatures, do you think it would be too much complexity to set non-blocking automatically based on whether it is CPU -> GPU transfer or GPU -> CPU transfer by inspecting the tensor's current device?

If you like the idea, perhaps this would be a different feature though, toggled by an accelerator flag?
body: @Chris-hughes10  CPU -> GPU and GPU -> CPU both lead to the same issues as mentioned above. Only GPU -> GPU is the safe operation but as I said above, it seems that it requires the two GPUs synchronization whether we set `non_blocking=True` or not
body: > @Chris-hughes10 CPU -> GPU and GPU -> CPU both lead to the same issues as mentioned above. Only GPU -> GPU is the safe operation but as I said above, it seems that it requires the two GPUs synchronization whether we set `non_blocking=True` or not

Apologies, I misread the post above. Please ignore my previous suggestion!