Repository: numba/numba

Issue:
Using a non-default device in CuPy leads to error in cuPointerGetAttribute
## Reporting a bug
- [x] I am using the latest released version of Numba (most recent is visible in
 the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [x] I have included below a minimal working reproducer (if you are unsure how
 to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

Not sure if I should report here or to CuPy, so please let me know, thanks.

When interoperating Numba with CuPy, I found that if a non-default device is used using CuPy's API, I got an error: `numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to cuPointerGetAttribute results in CUDA_ERROR_INVALID_VALUE` (see below). 

I think it's due to the detection of current CUDA context as handled by either party (possibly dup of  #3436?), because if I turned on either of the lines marked "workaround", the error goes away.

I am sure `cuda_array_interface` is in place. See the Numba and CuPy versions in the end. Also, as I said above, the MWE could work with the workaround, showing `cuda_array_interface` is indeed functioning.

MWE:
```python
from numba import cuda
import cupy as cp

@cuda.jit
def increment_by_one(an_array):
    pos = cuda.grid(1)
    if pos < an_array.size:
        an_array[pos] += 1

# set device; no error if we don't do this
cp.cuda.Device(3).use()     # set cupy device; I got 4 GPUs on this test node
#cuda.get_current_device()  # workaround: to avoid error, we need either this
#cuda.select_device(0)      # workaround: or this, but putting 0 instead of 3 still works, why?

# init arrays
an_array_in = cp.random.random(10)
an_array = an_array_in.copy()

# run
threadsperblock = 32
blockspergrid = (an_array.size + (threadsperblock - 1)) // threadsperblock
increment_by_one[blockspergrid, threadsperblock](an_array)
print(an_array - an_array_in)
```

Error message:
```shell
leofang@xfABCD-srv5:~/test$ python numba_cupy.py 
Traceback (most recent call last):
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/typing/context.py", line 344, in resolve_argument_type
    return typeof(val, Purpose.argument)
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/typing/typeof.py", line 34, in typeof
    raise ValueError(msg)
ValueError: cannot determine Numba type of <class 'cupy.core.core.ndarray'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "numba_cupy.py", line 45, in <module>
    increment_by_one[blockspergrid, threadsperblock](an_array)
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/cuda/compiler.py", line 765, in __call__
    kernel = self.specialize(*args)
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/cuda/compiler.py", line 775, in specialize
    [self.typingctx.resolve_argument_type(a) for a in args])
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/cuda/compiler.py", line 775, in <listcomp>
    [self.typingctx.resolve_argument_type(a) for a in args])
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/typing/context.py", line 347, in resolve_argument_type
    return typeof(numba.cuda.as_cuda_array(val), Purpose.argument)
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/cuda/api.py", line 60, in as_cuda_array
    owner=obj)
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/cuda/cudadrv/devices.py", line 212, in _require_cuda_context
    return fn(*args, **kws)
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/cuda/api.py", line 41, in from_cuda_array_interface
    devptr = driver.get_devptr_for_active_ctx(desc['data'][0])
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1689, in get_devptr_for_active_ctx
    driver.cuPointerGetAttribute(byref(devptr), attr, ptr)
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 290, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/home/leofang/conda_envs/cupy_dev/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 325, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to cuPointerGetAttribute results in CUDA_ERROR_INVALID_VALUE
```

Environment:
```shell
leofang@xfABCD-srv5:~/test$ numba -s
System info:
--------------------------------------------------------------------------------
__Time Stamp__
2019-05-16 03:58:59.738617

__Hardware Information__
Machine                                       : x86_64
CPU Name                                      : broadwell
CPU count                                     : 40
CFS restrictions                              : Information not available
CPU Features                                  : 
adx aes avx avx2 bmi bmi2 cmov cx16 f16c fma fsgsbase invpcid lzcnt mmx movbe
pclmul popcnt prfchw rdrnd rdseed rtm sahf sse sse2 sse3 sse4.1 sse4.2 ssse3
xsave xsaveopt

__OS Information__
Platform                                      : Linux-3.16.0-4-amd64-x86_64-with-debian-8.9
Release                                       : 3.16.0-4-amd64
System Name                                   : Linux
Version                                       : #1 SMP Debian 3.16.43-2+deb8u2 (2017-06-26)
OS specific info                              : debian8.9
glibc info                                    : glibc 2.9

__Python Information__
Python Compiler                               : GCC 7.2.0
Python Implementation                         : CPython
Python Version                                : 3.6.5
Python Locale                                 : en_US UTF-8

__LLVM information__
LLVM version                                  : 7.0.1

__CUDA Information__
Found 4 CUDA devices
id 0      b'Graphics Device'                              [SUPPORTED]
                      compute capability: 7.0
                           pci device id: 0
                              pci bus id: 2
id 1      b'Graphics Device'                              [SUPPORTED]
                      compute capability: 7.0
                           pci device id: 0
                              pci bus id: 3
id 2      b'Graphics Device'                              [SUPPORTED]
                      compute capability: 7.0
                           pci device id: 0
                              pci bus id: 130
id 3      b'Graphics Device'                              [SUPPORTED]
                      compute capability: 7.0
                           pci device id: 0
                              pci bus id: 131
Summary:
	4/4 devices are supported
CUDA driver version                           : 9010
CUDA libraries:
Finding cublas
	ERROR: can't locate lib
Finding cusparse
	ERROR: can't locate lib
Finding cufft
	ERROR: can't locate lib
Finding curand
	ERROR: can't locate lib
Finding nvvm
	named  libnvvm.so
	finding libdevice for compute_20...	ok
	finding libdevice for compute_30...	ok
	finding libdevice for compute_35...	ok
	finding libdevice for compute_50...	ok

__ROC Information__
ROC available                                 : False
Error initialising ROC due to                 : No ROC toolchains found.
No HSA Agents found, encountered exception when searching:
Error at driver init: 
NUMBA_HSA_DRIVER /opt/rocm/lib/libhsa-runtime64.so is not a valid file path.  Note it must be a filepath of the .so/.dll/.dylib or the driver:

__SVML Information__
SVML state, config.USING_SVML                 : False
SVML library found and loaded                 : False
llvmlite using SVML patched LLVM              : True
SVML operational                              : False

__Threading Layer Information__
TBB Threading layer available                 : False
+--> Disabled due to                          : Unknown import problem.
OpenMP Threading layer available              : False
+--> Disabled due to                          : Unknown import problem.
Workqueue Threading layer available           : True

__Numba Environment Variable Information__
None set.

__Conda Information__
conda_build_version                           : 3.0.9
conda_env_version                             : 4.3.29
platform                                      : linux-64
python_version                                : 3.5.1.final.0
root_writable                                 : False

__Current Conda Env__
alabaster                 0.7.12                    <pip>
anaconda-client           1.6.14                   py36_0  
appdirs                   1.4.3                     <pip>
appdirs                   1.4.3            py36h28b3542_0  
asn1crypto                0.24.0                   py36_0  
astropy                   3.0.3            py36h14c3975_2  
atomicwrites              1.1.5                    py36_0  
attrs                     18.1.0                   py36_0  
attrs                     17.4.0                    <pip>
automat                   0.7.0                    py36_0  
autopep8                  1.3.5                     <pip>
Babel                     2.6.0                     <pip>
backcall                  0.1.0                    py36_0  
blas                      1.0                         mkl  
bleach                    2.1.3                    py36_0  
bluesky                   1.3.3                    py36_0    https://pergamon.cs.nsls2.local:8443/conda/nsls2-tag
bokeh                     1.0.2                    py36_0    http://alexandria/conda/defaults
boltons                   17.0.0                   py36_0    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
ca-certificates           2018.03.07                    0  
cairo                     1.14.12              h8948797_3    http://alexandria/conda/defaults
certifi                   2018.4.16                py36_0  
cffi                      1.11.5           py36h9745a5d_0  
chardet                   3.0.4            py36h0f667ec_1  
click                     7.0                      py36_0    http://alexandria/conda/defaults
cloudpickle               0.6.1                    py36_0    http://alexandria/conda/defaults
clyent                    1.2.2            py36h7e57e65_1  
constantly                15.1.0           py36h28b3542_0  
cryptography              2.2.2            py36h14c3975_0  
cupy-cuda91               6.0.0rc1                  <pip>
cycler                    0.10.0           py36h93f1223_0  
cython                    0.28.5           py36hf484d3e_0  
cytoolz                   0.9.0.1          py36h14c3975_0  
dask                      1.0.0                    py36_0    http://alexandria/conda/defaults
dask-core                 1.0.0                    py36_0    http://alexandria/conda/defaults
databroker                0.11.1                   py36_0    https://pergamon.cs.nsls2.local:8443/conda/nsls2-tag
dbus                      1.13.2               h714fa37_1  
decorator                 4.3.0                    py36_0  
distributed               1.25.0                   py36_0    http://alexandria/conda/defaults
doct                      1.0.5            py36h572038c_0    https://pergamon.cs.nsls2.local:8443/conda/nsls2-tag
docutils                  0.14                      <pip>
entrypoints               0.2.3                    py36_2  
event-model               1.5.0                    py36_1    https://pergamon.cs.nsls2.local:8443/conda/nsls2-tag
expat                     2.2.6                he6710b0_0    http://alexandria/conda/defaults
fastrlock                 0.4                       <pip>
fftw                      3.3.8                h7b6447c_3    http://alexandria/conda/defaults
filestore                 0.8.0            py36hcfe6513_0    https://pergamon.cs.nsls2.local:8443/conda/nsls2-tag
flake8                    3.5.0                     <pip>
fontconfig                2.13.0               h9420a91_0    http://alexandria/conda/defaults
freetype                  2.9.1                h8a8886c_1    http://alexandria/conda/defaults
fribidi                   1.0.5                h7b6447c_0    http://alexandria/conda/defaults
glib                      2.56.1               h000015b_0  
glueviz                   0.9.1                    py36_0  
gmp                       6.1.2                h6c8ec71_1    http://alexandria/conda/defaults
graphite2                 1.3.12               h23475e2_2    http://alexandria/conda/defaults
graphviz                  2.40.1               h21bd128_2    http://alexandria/conda/defaults
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.9.0            py36h7918eee_0  
harfbuzz                  1.8.8                hffaf4a1_0    http://alexandria/conda/defaults
hdf5                      1.10.4               hb1b8bf9_0  
heapdict                  1.0.0                    py36_2    http://alexandria/conda/defaults
historydict               1.2.3                      py_0    https://pergamon.cs.nsls2.local:8443/conda/nsls2-tag
html5lib                  1.0.1                    py36_0  
humanize                  0.5.1                    py36_0  
hxntools                  0.4.1                    py36_0    https://pergamon.cs.nsls2.local:8443/conda/nsls2-tag
hyperlink                 18.0.0                   py36_0  
icu                       58.2                 h9c2bf20_1    http://alexandria/conda/defaults
idna                      2.6              py36h82fb2a8_1  
imagesize                 1.1.0                     <pip>
incremental               17.5.0                   py36_0  
intel-openmp              2018.0.0                      8  
ipykernel                 4.8.2                    py36_0  
ipython                   6.3.1                    py36_0  
ipython_genutils          0.2.0            py36hb52b0d5_0  
jedi                      0.12.0                   py36_1  
jinja2                    2.10             py36ha16c418_0  
jpeg                      9b                   h024ee3a_2    http://alexandria/conda/defaults
jsonschema                2.6.0            py36h006f8b5_0  
jupyter_client            5.2.3                    py36_0  
jupyter_core              4.4.0            py36h7c827e3_0  
jupyterlab                0.33.8                   py36_0  
jupyterlab_launcher       0.11.2                   py36_0  
kiwisolver                1.0.1            py36hf484d3e_0    http://alexandria/conda/defaults
libedit                   3.1                  heed3624_0  
libffi                    3.2.1                hd88cf55_4    http://alexandria/conda/defaults
libgcc                    7.2.0                h69d50b8_2  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.2.0                hdf63c60_3  
libpng                    1.6.34               hb9fc6fc_0  
libsodium                 1.0.16               h1bed415_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.9                he85c1e1_1  
libuuid                   1.0.3                h1bed415_2    http://alexandria/conda/defaults
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.8                hf84eae3_0  
line_profiler             2.1.2            py36h14c3975_0  
llvmlite                  0.28.0           py36hdbcaa40_0    conda-forge
lmfit                     0.9.7                    py36_0    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
locket                    0.2.0            py36h787c0ad_1    http://alexandria/conda/defaults
Mako                      1.0.7                     <pip>
markupsafe                1.0              py36hd9260cd_1  
MarkupSafe                1.0                       <pip>
matplotlib                2.2.3            py36h04863e7_2    http://alexandria/conda/nsls2-tag
mccabe                    0.6.1                     <pip>
metadatastore             0.7.0            py36h11c3279_0    https://pergamon.cs.nsls2.local:8443/conda/nsls2-tag
mistune                   0.8.3            py36h14c3975_1  
mkl                       2019.1                      144  
mkl-service               1.1.2            py36he904b0f_5  
mkl_fft                   1.0.10           py36ha843d7b_0  
mkl_random                1.0.2            py36hd81dba3_0  
mongoquery                1.1.0                    py36_0    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
more-itertools            4.2.0                    py36_0  
more-itertools            4.1.0                     <pip>
mpi                       1.0                     openmpi    http://alexandria/conda/nsls2-tag
mpi4py                    3.0.0           py36_openmpi_1004    http://alexandria/conda/nsls2-tag
msgpack-python            0.5.6            py36h6bb024c_1    http://alexandria/conda/defaults
nbconvert                 5.3.1                    py36_0  
nbformat                  4.4.0            py36h31c9010_0  
ncurses                   6.0                  h9df7e31_2    http://alexandria/conda/defaults
networkx                  2.1                      py36_0  
notebook                  5.6.0                    py36_0  
numba                     0.43.1           py36hf2d7682_0    conda-forge
numpy                     1.15.4           py36h7e9f1db_0  
numpy-base                1.15.4           py36hde5b4d6_0  
olefile                   0.45.1                   py36_0  
openmpi                   3.1.1                      1000    http://alexandria/conda/nsls2-tag
openssl                   1.0.2o               h20670df_0  
ophyd                     1.0.0                    py36_0    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
packaging                 18.0                     py36_0    http://alexandria/conda/defaults
packaging                 19.0                      <pip>
pandas                    0.23.0           py36h637b7d7_0  
pandoc                    2.2.1                h629c226_0  
pandocfilters             1.4.2                    py36_1  
pango                     1.42.4               h049681c_0    http://alexandria/conda/defaults
parso                     0.2.0                    py36_0  
partd                     0.3.9                    py36_0    http://alexandria/conda/defaults
pbr                       4.0.4                     <pip>
pcre                      8.42                 h439df22_0  
pexpect                   4.5.0                    py36_0  
pickleshare               0.7.4            py36h63277f8_0  
pillow                    5.3.0            py36h34e0f95_0    http://alexandria/conda/defaults
pims                      0.4                      py36_0    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
pip                       9.0.3                    py36_0  
pixman                    0.34.0               hceecf20_3    http://alexandria/conda/defaults
pluggy                    0.6.0                     <pip>
pluggy                    0.6.0            py36hb689045_0  
posix_ipc                 1.0.4            py36h14c3975_0  
prettytable               0.7.2                    py36_1    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
prometheus_client         0.3.1                    py36_0  
prompt_toolkit            1.0.15           py36h17d85b1_0  
psutil                    5.4.5            py36h14c3975_0  
ptyprocess                0.5.2            py36h69acd42_0  
py                        1.5.3                    py36_0  
py                        1.5.3                     <pip>
pyasn1                    0.4.4                    py36_0  
pyasn1-modules            0.2.2                    py36_0  
pycodestyle               2.3.1                     <pip>
pycparser                 2.18             py36hf9f622e_1  
pycuda                    2017.1.1                  <pip>
pyepics                   3.2.6                    py36_0    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
pyfftw                    0.11.1                   py36_0    http://alexandria/conda/nsls2-tag
pyFFTW                    0.10.4+227.g701350d           <pip>
pyflakes                  1.6.0                     <pip>
pygments                  2.2.0            py36h0d3125c_0  
pymongo                   3.4.0                    py36_0  
pyopenssl                 17.5.0           py36h20ba746_0  
pyparsing                 2.2.0            py36hee85983_1  
pyqt                      5.9.2            py36h751905a_0  
pysocks                   1.6.8                    py36_0  
pytest                    3.6.0                    py36_1  
pytest                    3.5.1                     <pip>
pytest-arraydiff          0.2                      py36_0  
pytest-astropy            0.4.0                    py36_0  
pytest-doctestplus        0.1.3                    py36_0  
pytest-openfiles          0.3.0                    py36_0  
pytest-remotedata         0.3.0                    py36_0  
python                    3.6.5                hc3d631a_0  
python-dateutil           2.7.2                    py36_0  
python-graphviz           0.8.4                    py36_1    http://alexandria/conda/defaults
pytools                   2018.4                    <pip>
pytz                      2018.4                   py36_0  
pyyaml                    3.12             py36hafb9ca4_1  
pyzmq                     17.0.0           py36h14c3975_0  
qt                        5.9.6                h52aff34_0    http://alexandria/conda/defaults
qtpy                      1.4.2                    py36_0  
readline                  7.0                  ha6073c6_4    http://alexandria/conda/defaults
requests                  2.18.4           py36he2e5f8d_1  
scikit-cuda               0.5.1                     <pip>
scipy                     1.1.0            py36h7c811a0_2  
send2trash                1.5.0                    py36_0  
service_identity          17.0.0           py36h28b3542_0  
setuptools                39.0.1                   py36_0  
simplegeneric             0.8.1                    py36_2  
sip                       4.19.8           py36hf484d3e_0  
six                       1.11.0           py36h372c433_1  
slicerator                0.9.8                    py36_0    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
snowballstemmer           1.2.1                     <pip>
sortedcontainers          2.1.0                    py36_0    http://alexandria/conda/defaults
Sphinx                    1.8.4                     <pip>
sphinx-rtd-theme          0.4.3                     <pip>
sphinxcontrib-websupport  1.1.0                     <pip>
sqlite                    3.23.1               he433501_0  
super_state_machine       1.0              py36h7bccc9c_0    https://pergamon.cs.nsls2.local:8443/conda/nsls2-tag
tblib                     1.3.2                    py36_0    http://alexandria/conda/defaults
terminado                 0.8.1                    py36_1  
testpath                  0.3.1                    py36_0  
tifffile                  0.12.1                   py36_1    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
tk                        8.6.8                hbc83047_0    http://alexandria/conda/defaults
toolz                     0.9.0                    py36_0  
tornado                   5.0.2                    py36_0  
tqdm                      4.23.4                   py36_0  
traitlets                 4.3.2            py36h674d592_0  
twisted                   18.7.0           py36h14c3975_1  
tzlocal                   1.4                      py36_0    https://pergamon.cs.nsls2.local:8443/conda/conda-forge
ujson                     1.35                     py36_0  
urllib3                   1.22             py36hbe7ace6_0  
wcwidth                   0.1.7            py36hdf4376a_0  
webencodings              0.5.1                    py36_1  
wheel                     0.31.0                   py36_0  
xz                        5.2.4                h14c3975_4    http://alexandria/conda/defaults
yaml                      0.1.7                had09818_2    http://alexandria/conda/defaults
zeromq                    4.2.5                h439df22_0  
zict                      0.1.3                    py36_0    http://alexandria/conda/defaults
zlib                      1.2.11               h7b6447c_3    http://alexandria/conda/defaults
zope                      1.0                      py36_0  
zope.interface            4.5.0            py36h14c3975_0  
--------------------------------------------------------------------------------
```


Discussion:
@sklam: This looks like something we were worried about early on with the CuPy support.
@leofang, it would be useful to know the topology of your GPU system.  I suspect this could be related to whether the devices can be peer-accessing each other.  For instance, GPUs on different PCIe domain cannot peer-access each other.  Are there pairs of GPUs that your MWE will work? (Am I right that your 2nd workaround imply that device 0 works with device 3?)
> Are there pairs of GPUs that your MWE will work? (Am I right that your 2nd workaround imply that device 0 works with device 3?)

@sklam, I don't think it's related to topology. It's more like a bug in Numba's way of looking up devices.
In my second workaround, I meant to say that I could choose any device (0, 1, 2, or 3) there and it'll work. But this is not the expected behavior: if the device context matters, only `cuda.select_device(3)` can work, and 0, 1, and 2 should fail. 

Moreover, in my example I meant to work with a single device. I just wanna examine the ability of choosing a particular one, so topology can and should be neglected.

My topology is already posted in the environment info, see the pci bus ids. For clarity I'm reposting below, and I also post `nvidia-smi` message:
```shell
__CUDA Information__
Found 4 CUDA devices
id 0      b'Graphics Device'                              [SUPPORTED]
                      compute capability: 7.0
                           pci device id: 0
                              pci bus id: 2
id 1      b'Graphics Device'                              [SUPPORTED]
                      compute capability: 7.0
                           pci device id: 0
                              pci bus id: 3
id 2      b'Graphics Device'                              [SUPPORTED]
                      compute capability: 7.0
                           pci device id: 0
                              pci bus id: 130
id 3      b'Graphics Device'                              [SUPPORTED]
                      compute capability: 7.0
                           pci device id: 0
                              pci bus id: 131
Summary:
	4/4 devices are supported
```
```shell
leofang@xfABCD-srv5:~/$ nvidia-smi topo -m
	GPU0	GPU1	GPU2	GPU3	CPU Affinity
GPU0	 X 	PHB	SYS	SYS	0-9,20-29
GPU1	PHB	 X 	SYS	SYS	0-9,20-29
GPU2	SYS	SYS	 X 	PHB	10-19,30-39
GPU3	SYS	SYS	PHB	 X 	10-19,30-39

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing a single PCIe switch
  NV#  = Connection traversing a bonded set of # NVLinks
```
> In my second workaround, I meant to say that I could choose any device (0, 1, 2, or 3) there and it'll work. 

This part is really odd. 

 I need find a similar machine to test it.  On my 2-GPU machine:

```
$ nvidia-smi topo -m
	GPU0	GPU1	CPU Affinity
GPU0	 X 	PHB	0-7
GPU1	PHB	 X 	0-7
```
it is always working.

Btw, the error is triggered from a call to the cuda driver API to get a device-pointer usable by the active  CUDA context.  The doc on the error code says:

> If there exists no device pointer value through which kernels running in the current CUcontext may access ptr then CUDA_ERROR_INVALID_VALUE is returned.

(from https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__UNIFIED.html#group__CUDA__UNIFIED_1g0c28ed0aff848042bc0533110e45820c)

It is unclear what can be causing the error though.

I see that numba is overriding what cupy has set as the active device.  That is why it is working when you select device with `numba.cuda.select_device(NN)`.  You can verify this by printing `an_array_in.device`.  Even `numba.cuda.get_current_device()` is attaching to device 0.  If you skip `cp.cuda.Device(3).use()`, cupy is able to use the device attached by numba.  

Running the MWE as is, cupy will allocate on device 3 but numba will try to read the array on device 0.  Since the devices are on different PCIe domain, peer-access is not available and `cuPointerGetAttribute ` fail.

What needs to fix:

* Numba needs to respect active CUDA context attached by thirdparty and not to override it implicitly.
* Numba handled similar peer-inaccessible problems on CUDA IPC side automatically.  We should consider handling it automatically for `__cuda_array_interface__` as well.


Resolving patch:
diff --git a/numba/cuda/cudadrv/devices.py b/numba/cuda/cudadrv/devices.py
--- a/numba/cuda/cudadrv/devices.py
+++ b/numba/cuda/cudadrv/devices.py
@@ -13,7 +13,8 @@
 from __future__ import print_function, absolute_import, division
 import functools
 import threading
-from numba import servicelib
+from contextlib import contextmanager
+
 from .driver import driver
 
 
@@ -52,8 +53,10 @@ def __len__(self):
     def current(self):
         """Returns the active device or None if there's no active device
         """
-        if _runtime.context_stack:
-            return self.lst[_runtime.current_context.device.id]
+        with driver.get_active_context() as ac:
+            devnum = ac.devnum
+            if devnum is not None:
+                return self[devnum]
 
 
 class _DeviceContextManager(object):
@@ -75,10 +78,11 @@ def __getattr__(self, item):
         return getattr(self._device, item)
 
     def __enter__(self):
-        _runtime.push_context(self)
+        _runtime.get_or_create_context(self._device.id)
 
     def __exit__(self, exc_type, exc_val, exc_tb):
-        _runtime.pop_context()
+        # this will verify that we are popping the right device context.
+        self._device.get_primary_context().pop()
 
     def __str__(self):
         return "<Managed Device {self.id}>".format(self=self)
@@ -94,8 +98,8 @@ class _Runtime(object):
     def __init__(self):
         self.gpus = _DeviceList()
 
-        # A thread local stack
-        self.context_stack = servicelib.TLStack()
+        # For caching the attached CUDA Context
+        self._tls = threading.local()
 
         # Remember the main thread
         # Only the main thread can *actually* destroy
@@ -104,71 +108,84 @@ def __init__(self):
         # Avoid mutation of runtime state in multithreaded programs
         self._lock = threading.RLock()
 
-    @property
-    def current_context(self):
-        """Return the active gpu context
-        """
-        return self.context_stack.top
+    @contextmanager
+    def ensure_context(self):
+        """Ensure a CUDA context is available inside the context.
 
-    def _get_or_create_context(self, gpu):
-        """Try to use a already created context for the given gpu.  If none
-        existed, create a new context.
+        On entrance, queries the CUDA driver for an active CUDA context and
+        attaches it in TLS for subsequent calls so they do not need to query
+        the CUDA driver again.  On exit, detach the CUDA context from the TLS.
 
-        Returns the context
+        This will allow us to pickup thirdparty activated CUDA context in
+        any top-level Numba CUDA API.
         """
-        with self._lock:
-            ctx = gpu.get_primary_context()
-            ctx.push()
-            return ctx
+        with driver.get_active_context():
+            oldctx = self._get_attached_context()
+            newctx = self.get_or_create_context(None)
+            self._set_attached_context(newctx)
+            try:
+                yield
+            finally:
+                self._set_attached_context(oldctx)
 
-    def push_context(self, gpu):
-        """Push a context for the given GPU or create a new one if no context
-        exist for the given GPU.
+    def get_or_create_context(self, devnum):
+        """Returns the primary context and push+create it if needed
+        for *devnum*.  If *devnum* is None, use the active CUDA context (must
+        be primary) or create a new one with ``devnum=0``.
         """
-        # Context stack is empty or the active device is not the given gpu
-        if self.context_stack.is_empty or self.current_context.device != gpu:
-            ctx = self._get_or_create_context(gpu)
-
-        # Active context is from the gpu
+        if devnum is None:
+            attached_ctx = self._get_attached_context()
+            if attached_ctx is None:
+                return self._get_or_create_context_uncached(devnum)
+            else:
+                return attached_ctx
         else:
-            ctx = self.current_context
-
-        # Always put the new context on the stack
-        self.context_stack.push(ctx)
-        return ctx
+            return self._activate_context_for(devnum)
 
-    def pop_context(self):
-        """Pop a context from the context stack if there is more than
-        one context in the stack.
-
-        Will not remove the last context in the stack.
+    def _get_or_create_context_uncached(self, devnum):
+        """See also ``get_or_create_context(devnum)``.
+        This version does not read the cache.
         """
-        ctx = self.current_context
-        # If there is more than one context
-        # Do not pop the last context so there is always a active context
-        if len(self.context_stack) > 1:
-            ctx.pop()
-            self.context_stack.pop()
-        assert self.context_stack
+        with self._lock:
+            # Try to get the active context in the CUDA stack or
+            # activate GPU-0 with the primary context
+            with driver.get_active_context() as ac:
+                if not ac:
+                    return self._activate_context_for(0)
+                else:
+                    # Get primary context for the active device
+                    ctx = self.gpus[ac.devnum].get_primary_context()
+                    # Is active context the primary context?
+                    if ctx.handle.value != ac.context_handle.value:
+                        msg = ('Numba cannot operate on non-primary'
+                               ' CUDA context {:x}')
+                        raise RuntimeError(msg.format(ac.context_handle.value))
+                return ctx
+
+    def _activate_context_for(self, devnum):
+        with self._lock:
+            gpu = self.gpus[devnum]
+            newctx = gpu.get_primary_context()
+            # Detect unexpected context switch
+            cached_ctx = self._get_attached_context()
+            if cached_ctx is not None and cached_ctx is not newctx:
+                raise RuntimeError('Cannot switch CUDA-context.')
+            newctx.push()
+            return newctx
 
-    def get_or_create_context(self, devnum):
-        """Returns the current context or push/create a context for the GPU
-        with the given device number.
-        """
-        if self.context_stack:
-            return self.current_context
-        else:
-            with self._lock:
-                return self.push_context(self.gpus[devnum])
+    def _get_attached_context(self):
+        return getattr(self._tls, 'attached_context', None)
+
+    def _set_attached_context(self, ctx):
+        self._tls.attached_context = ctx
 
     def reset(self):
         """Clear all contexts in the thread.  Destroy the context if and only
         if we are in the main thread.
         """
-        # Clear the context stack
-        while self.context_stack:
-            ctx = self.context_stack.pop()
-            ctx.pop()
+        # Pop all active context.
+        while driver.pop_active_context() is not None:
+            pass
 
         # If it is the main thread
         if threading.current_thread() == self._mainthread:
@@ -187,7 +204,7 @@ def _destroy_all_contexts(self):
 gpus = _runtime.gpus
 
 
-def get_context(devnum=0):
+def get_context(devnum=None):
     """Get the current device or use a device by device number, and
     return the CUDA context.
     """
@@ -198,18 +215,12 @@ def require_context(fn):
     """
     A decorator that ensures a CUDA context is available when *fn* is executed.
 
-    Decorating *fn* is equivalent to writing::
-
-       get_context()
-       fn()
-
-    at each call site.
+    Note: The function *fn* cannot switch CUDA-context.
     """
-
     @functools.wraps(fn)
     def _require_cuda_context(*args, **kws):
-        get_context()
-        return fn(*args, **kws)
+        with _runtime.ensure_context():
+            return fn(*args, **kws)
 
     return _require_cuda_context
 
diff --git a/numba/cuda/cudadrv/driver.py b/numba/cuda/cudadrv/driver.py
--- a/numba/cuda/cudadrv/driver.py
+++ b/numba/cuda/cudadrv/driver.py
@@ -20,6 +20,7 @@
 import copy
 import warnings
 import logging
+import threading
 from itertools import product
 from ctypes import (c_int, byref, c_size_t, c_char, c_char_p, addressof,
                     c_void_p, c_float)
@@ -350,15 +351,67 @@ def reset(self):
         for dev in self.devices.values():
             dev.reset()
 
-    def get_context(self):
-        """Get current active context in CUDA driver runtime.
-        Note: Lowlevel calls that returns the handle.
+    def pop_active_context(self):
+        """Pop the active CUDA context and return the handle.
+        If no CUDA context is active, return None.
         """
-        handle = drvapi.cu_context(0)
-        driver.cuCtxGetCurrent(byref(handle))
-        if not handle.value:
-            return None
-        return handle
+        with self.get_active_context() as ac:
+            if ac.devnum is not None:
+                popped = drvapi.cu_context()
+                driver.cuCtxPopCurrent(byref(popped))
+                return popped
+
+    def get_active_context(self):
+        """Returns an instance of ``_ActiveContext``.
+        """
+        return _ActiveContext()
+
+
+class _ActiveContext(object):
+    """An contextmanager object to cache active context to reduce dependency
+    on querying the CUDA driver API.
+
+    Once entering the context, it is assumed that the active CUDA context is
+    not changed until the context is exited.
+    """
+    _tls_cache = threading.local()
+
+    def __enter__(self):
+        is_top = False
+        # check TLS cache
+        if hasattr(self._tls_cache, 'ctx_devnum'):
+            hctx, devnum = self._tls_cache.ctx_devnum
+        # Not cached. Query the driver API.
+        else:
+            hctx = drvapi.cu_context(0)
+            driver.cuCtxGetCurrent(byref(hctx))
+            hctx = hctx if hctx.value else None
+
+            if hctx is None:
+                devnum = None
+            else:
+                hdevice = drvapi.cu_device()
+                driver.cuCtxGetDevice(byref(hdevice))
+                devnum = hdevice.value
+
+                self._tls_cache.ctx_devnum = (hctx, devnum)
+                is_top = True
+
+        self._is_top = is_top
+        self.context_handle = hctx
+        self.devnum = devnum
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        if self._is_top:
+            delattr(self._tls_cache, 'ctx_devnum')
+
+    def __bool__(self):
+        """Returns True is there's a valid and active CUDA context.
+        """
+        return self.context_handle is not None
+
+    __nonzero__ = __bool__
 
 
 driver = Driver()
@@ -605,13 +658,13 @@ def reset(self):
         """
         Clean up all owned resources in this context.
         """
-        _logger.info('reset context of device %s', self.device.id)
         # Free owned resources
         _logger.info('reset context of device %s', self.device.id)
         self.allocations.clear()
         self.modules.clear()
         # Clear trash
-        self.deallocations.clear()
+        if self.deallocations:
+            self.deallocations.clear()
 
     def get_memory_info(self):
         """Returns (free, total) memory in bytes in the context.
@@ -669,8 +722,7 @@ def pop(self):
         Pops this context off the current CPU thread. Note that this context must
         be at the top of the context stack, otherwise an error will occur.
         """
-        popped = drvapi.cu_context()
-        driver.cuCtxPopCurrent(byref(popped))
+        popped = driver.pop_active_context()
         assert popped.value == self.handle.value
 
     def _attempt_allocation(self, allocator):


Test patch:
diff --git a/numba/cuda/tests/cudadrv/test_context_stack.py b/numba/cuda/tests/cudadrv/test_context_stack.py
--- a/numba/cuda/tests/cudadrv/test_context_stack.py
+++ b/numba/cuda/tests/cudadrv/test_context_stack.py
@@ -1,9 +1,12 @@
 from __future__ import print_function
 
 import numbers
+from ctypes import byref
+import weakref
 
 from numba import cuda
-from numba.cuda.testing import unittest, SerialMixin
+from numba.cuda.testing import unittest, SerialMixin, skip_on_cudasim
+from numba.cuda.cudadrv import driver
 
 
 class TestContextStack(SerialMixin, unittest.TestCase):
@@ -26,6 +29,9 @@ def test_gpus_iter(self):
 
 class TestContextAPI(SerialMixin, unittest.TestCase):
 
+    def tearDown(self):
+        cuda.close()
+
     def test_context_memory(self):
         mem = cuda.current_context().get_memory_info()
 
@@ -37,6 +43,69 @@ def test_context_memory(self):
 
         self.assertLessEqual(mem.free, mem.total)
 
+    @unittest.skipIf(len(cuda.gpus) < 2, "need more than 1 gpus")
+    @skip_on_cudasim('CUDA HW required')
+    def test_forbidden_context_switch(self):
+        # Cannot switch context inside a `cuda.require_context`
+        @cuda.require_context
+        def switch_gpu():
+            with cuda.gpus[1]:
+                pass
+
+        with cuda.gpus[0]:
+            with self.assertRaises(RuntimeError) as raises:
+                switch_gpu()
+
+            self.assertIn("Cannot switch CUDA-context.", str(raises.exception))
+
+    @unittest.skipIf(len(cuda.gpus) < 2, "need more than 1 gpus")
+    def test_accepted_context_switch(self):
+        def switch_gpu():
+            with cuda.gpus[1]:
+                return cuda.current_context().device.id
+
+        with cuda.gpus[0]:
+            devid = switch_gpu()
+        self.assertEqual(devid, 1)
+
+
+@skip_on_cudasim('CUDA HW required')
+class Test3rdPartyContext(SerialMixin, unittest.TestCase):
+    def tearDown(self):
+        cuda.close()
+
+    def test_attached_primary(self):
+        # Emulate primary context creation by 3rd party
+        the_driver = driver.driver
+        hctx = driver.drvapi.cu_context()
+        the_driver.cuDevicePrimaryCtxRetain(byref(hctx), 0)
+        try:
+            ctx = driver.Context(weakref.proxy(self), hctx)
+            ctx.push()
+            # Check that the context from numba matches the created primary
+            # context.
+            my_ctx = cuda.current_context()
+            self.assertEqual(my_ctx.handle.value, ctx.handle.value)
+        finally:
+            ctx.pop()
+            the_driver.cuDevicePrimaryCtxRelease(0)
+
+    def test_attached_non_primary(self):
+        # Emulate non-primary context creation by 3rd party
+        the_driver = driver.driver
+        hctx = driver.drvapi.cu_context()
+        the_driver.cuCtxCreate(byref(hctx), 0, 0)
+        try:
+            cuda.current_context()
+        except RuntimeError as e:
+            # Expecting an error about non-primary CUDA context
+            self.assertIn("Numba cannot operate on non-primary CUDA context ",
+                          str(e))
+        else:
+            self.fail("No RuntimeError raised")
+        finally:
+            the_driver.cuCtxDestroy(hctx)
+
 
 if __name__ == '__main__':
     unittest.main()