Repository: cupy/cupy

Issue:
Stream in the context-manager form is not used in `ElementwiseKernel` or `ReductionKernel`
This is actually a bug reported back in #1695 that unfortunately went unnoticed. 

In `examples/stream/map_reduce.py`, a list of streams was created for executing `cupy.matmul()` in parallel, which is backed by a `ReductionKernel` in this case: https://github.com/cupy/cupy/blob/1af22f57fda92ae35bde806d0c4d110faf4fed52/cupy/core/core.pyx#L2513-L2516
However, inspecting the implementation I found that `ReductionKernel` only accepts an explicit `stream` argument; it does not pick up any current stream: https://github.com/cupy/cupy/blob/32718607a7808ec6bc3a24cf9231a9351f8fc95e/cupy/core/reduction.pxi#L396
In other words, that example was misleading because those streams were not used at all and so all executions were serialized, as can be checked from nvprof + nvvp (see the circle in red):
<img width="972" alt="螢幕快照 2019-10-03 上午11 24 27" src="https://user-images.githubusercontent.com/5534781/66140715-978bf180-e5d0-11e9-8228-f613a3eba6fd.png">

The same bug also appears in `ElementwiseKernel`:
https://github.com/cupy/cupy/blob/1af22f57fda92ae35bde806d0c4d110faf4fed52/cupy/core/_kernel.pyx#L537

In my opinion, unlike `RawKernel` which is not used by any CuPy core functionalities, `ElementwiseKernel` and `ReductionKernel` should honor the current stream by checking the current stream pointer if no stream argument is explicitly given, since many CuPy functions like `cupy.matmul()` do not support passing in a stream. A similar approach is already adopted in the FFT module, see #2362.


Discussion:
This issue is related to Item 1 of #2519.
Apparently, if the `stream` is set to None in those two functions, when the kernel is launched
the current stream is retrieved:
https://github.com/cupy/cupy/blob/32718607a7808ec6bc3a24cf9231a9351f8fc95e/cupy/cuda/function.pyx#L126

This is done in the `linear_launch` function
https://github.com/cupy/cupy/blob/32718607a7808ec6bc3a24cf9231a9351f8fc95e/cupy/cuda/function.pyx#L174
That's true, otherwise we'd see all computations collapsing on the null stream in nvvp...Not sure what I was thinking, but I'll need to investigate the reason of slowness further later. Let me first close this issue, and reopen if necessary.

Thank you for digging it up for me, @emcastillo! 
If in the map reduce example, you decouple the input initialization from the actual computation, you can see that the curand kernels are serialized, but the matrix dot ones overlap in the timeline.

```python
inputs=[]
for stream in map_streams:
    with stream:
        x = rand.normal(size=(1, 1024 * 256))
        y = rand.normal(size=(1024 * 256, 1))
        inputs.append((x, y))

# Map
for stream, inputs in zip(map_streams, inputs):
    with stream:
        x, y = inputs
        z = cupy.matmul(x, y)
        zs.append(z)
    stop_event = stream.record()
    stop_events.append(stop_event)
```
Actually, the previous code was fine, just increasing the matrix sizes shows real overlap
![image](https://user-images.githubusercontent.com/15824382/66288449-40f61e80-e914-11e9-9a33-7cc672499938.png)

Ah, I also suspected this but haven't had time to confirm it. Thanks a lot @emcastillo! Should I create a PR to improve that test? 

Resolving patch:
diff --git a/examples/stream/map_reduce.py b/examples/stream/map_reduce.py
--- a/examples/stream/map_reduce.py
+++ b/examples/stream/map_reduce.py
@@ -19,8 +19,8 @@
 # Map
 for stream in map_streams:
     with stream:
-        x = rand.normal(size=(1, 1024 * 256))
-        y = rand.normal(size=(1024 * 256, 1))
+        x = rand.normal(size=(1, 1024**2))
+        y = rand.normal(size=(1024**2, 1))
         z = cupy.matmul(x, y)
         zs.append(z)
     stop_event = stream.record()


Test patch: