Repository: cupy/cupy

Issue:
Device-to-host copy in `examples/stream/cupy_memcpy.py` is not asynchronous
I've interested in asynchronous memcpy for better performance. I checked the `cupy_memcpy.py` sample, and noticed that the last line `x_pinned_cpu = x_gpu.get()` [1] shouldn't work asynchronously. Unfortunately I don't know how to properly fix it.

One issue is trivial: this line re-binds the variable `x_pinned_cpu`, instead of updating the value bound to this variable.

But there is another more tricky problem. The function `cupy.ndarray.get` creates the new `numpy.ndarray` instance by calling `numpy.empty`, and passes the new host pointer to the `copy_to_host_async` function [2]. IIUC, as the new array not allocated in pinned memory, the copy couldn't be asynchronous with other computations at the GPU.

* [1] https://github.com/cupy/cupy/blob/v5.0.0rc1/examples/stream/cupy_memcpy.py#L24
* [2] https://github.com/cupy/cupy/blob/v5.0.0rc1/cupy/core/core.pyx#L1805


Discussion:
> The function cupy.ndarray.get creates the new numpy.ndarray instance by calling numpy.empty, and passes the new host pointer to the copy_to_host_async function [2]. IIUC, as the new array not allocated in pinned memory, the copy couldn't be asynchronous with other computations at the GPU.

~I thought that this is by design as ~it's not possible to construct numpy ndarray from pinned memory~ it's difficult to decide when to free the pinned memory.~ Currently `copy_to_host_async` is used just to run the memcpy on the user-specified stream.
@okuta do you have any follow up?

Resolving patch:
diff --git a/examples/stream/cupy_memcpy.py b/examples/stream/cupy_memcpy.py
--- a/examples/stream/cupy_memcpy.py
+++ b/examples/stream/cupy_memcpy.py
@@ -13,12 +13,38 @@ def _pin_memory(array):
     return ret
 
 
-x_cpu = numpy.array([1, 2, 3], dtype=numpy.float32)
-x_pinned_cpu = _pin_memory(x_cpu)
-x_gpu = cupy.core.ndarray((3,), dtype=numpy.float32)
-with cupy.cuda.stream.Stream():
-    x_gpu.set(x_pinned_cpu)
-
-stream = cupy.cuda.stream.Stream()
-stream.use()
-x_pinned_cpu = x_gpu.get()
+SIZE = 1024 * 1024
+x_cpu_src = numpy.arange(SIZE, dtype=numpy.float32)
+x_gpu_src = cupy.arange(SIZE, dtype=numpy.float32)
+
+
+# synchronous
+stream = cupy.cuda.Stream.null
+start = stream.record()
+x_gpu_dst = cupy.empty(x_cpu_src.shape, x_cpu_src.dtype)
+x_gpu_dst.set(x_cpu_src)
+x_cpu_dst = x_gpu_src.get()
+end = stream.record()
+
+print('Synchronous Device to Host / Host to Device (ms)')
+print(cupy.cuda.get_elapsed_time(start, end))
+
+
+# asynchronous
+x_gpu_dst = cupy.empty(x_cpu_src.shape, x_cpu_src.dtype)
+x_cpu_dst = numpy.empty(x_gpu_src.shape, x_gpu_src.dtype)
+
+x_pinned_cpu_src = _pin_memory(x_cpu_src)
+x_pinned_cpu_dst = _pin_memory(x_cpu_dst)
+
+with cupy.cuda.stream.Stream() as stream_htod:
+    start = stream_htod.record()
+    x_gpu_dst.set(x_pinned_cpu_src)
+    with cupy.cuda.stream.Stream() as stream_dtoh:
+        x_gpu_src.get(out=x_pinned_cpu_dst)
+        stream_dtoh.synchronize()
+    stream_htod.synchronize()
+    end = stream_htod.record()
+
+print('Asynchronous Device to Host / Host to Device (ms)')
+print(cupy.cuda.get_elapsed_time(start, end))


Test patch: