Repository: explosion/spaCy
Pull request: Add warning about GPU selection in Jupyter notebooks


<!--- Provide a general summary of your changes in the title. -->

## Description
<!--- Use this section to describe your changes. If your changes required
testing, include information about the testing environment and the tests you
ran. If your test fixes a bug reported in an issue, don't forget to include the
issue number. If your PR is still a work in progress, that's totally fine – just
include a note to let us know. -->

Due to the bug reported in #6990, warn users with `cupy` available who are working in a Jupyter notebook that they need to run `require_gpu` or `prefer_gpu` in the same cell as `spacy.load` for the model to be loaded on the correct device.

`thinc` uses `contextvars` to store the GPU selection and there is a bug in IPython (https://github.com/ipython/ipython/issues/11565) that does not preserve `contextvars` settings across cells.

### Types of change
<!-- What type of change does your PR cover? Is it a bug fix, an enhancement
or new feature, or a change to the documentation? -->

Enhancement?

## Checklist
<!--- Before you submit the PR, go over this checklist and make sure you can
tick off all the boxes. [] -> [x] -->
- [x] I have submitted the spaCy Contributor Agreement.
- [x] I ran the tests, and all new and existing tests passed.
- [x] My changes don't require a change to the documentation, or if they do, I've added all required information.


Problem statement:
Error when GPU initialization and model loading are in different notebook cells
## How to reproduce the behaviour
<!-- Include a code example or the steps that led to the problem. Please try to be as specific as possible. -->
Installed the latest version following the instruction
```
pip install -U pip setuptools wheel
pip install -U spacy[cuda110]
python -m spacy download en_core_web_trf
```
The installation was successful, and gpu is available:
```
import spacy
from thinc.api import set_gpu_allocator, require_gpu
import spacy_transformers
print(spacy.prefer_gpu(),require_gpu(0))
True True
```

However, Error: "a bytes-like object is required, not 'cupy.core.core.ndarray'" happened, when using "en_core_web_trf".
```
nlp = spacy.load("en_core_web_trf")
sent = 'Hello World'
doc = nlp(sent)
```

```
TypeError                                 Traceback (most recent call last)
<ipython-input-12-66e94dc9d1fd> in <module>
      1 sent = 'Hello World'
----> 2 doc = nlp(sent)


~/anaconda3/envs/acl/lib/python3.8/site-packages/spacy/language.py in __call__(self, text, disable, component_cfg)
    992                 raise ValueError(Errors.E109.format(name=name)) from e
    993             except Exception as e:
--> 994                 error_handler(name, proc, [doc], e)
    995             if doc is None:
    996                 raise ValueError(Errors.E005.format(name=name))

~/anaconda3/envs/acl/lib/python3.8/site-packages/spacy/util.py in raise_error(proc_name, proc, docs, e)
   1493 
   1494 def raise_error(proc_name, proc, docs, e):
-> 1495     raise e
   1496 
   1497 

~/anaconda3/envs/acl/lib/python3.8/site-packages/spacy/language.py in __call__(self, text, disable, component_cfg)
    987                 error_handler = proc.get_error_handler()
    988             try:
--> 989                 doc = proc(doc, **component_cfg.get(name, {}))
    990             except KeyError as e:
    991                 # This typically happens if a component is not initialized

~/anaconda3/envs/acl/lib/python3.8/site-packages/spacy/pipeline/trainable_pipe.pyx in spacy.pipeline.trainable_pipe.TrainablePipe.__call__()

~/anaconda3/envs/acl/lib/python3.8/site-packages/spacy/util.py in raise_error(proc_name, proc, docs, e)
   1493 
   1494 def raise_error(proc_name, proc, docs, e):
-> 1495     raise e
   1496 
   1497 

~/anaconda3/envs/acl/lib/python3.8/site-packages/spacy/pipeline/trainable_pipe.pyx in spacy.pipeline.trainable_pipe.TrainablePipe.__call__()

~/anaconda3/envs/acl/lib/python3.8/site-packages/spacy/pipeline/tagger.pyx in spacy.pipeline.tagger.Tagger.predict()

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/model.py in predict(self, X)
    310         only the output, instead of the `(output, callback)` tuple.
    311         """
--> 312         return self._func(self, X, is_train=False)[0]
    313 
    314     def finish_update(self, optimizer: Optimizer) -> None:

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/layers/chain.py in forward(model, X, is_train)
     52     callbacks = []
     53     for layer in model.layers:
---> 54         Y, inc_layer_grad = layer(X, is_train=is_train)
     55         callbacks.append(inc_layer_grad)
     56         X = Y

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/model.py in __call__(self, X, is_train)
    286         """Call the model's `forward` function, returning the output and a
    287         callback to compute the gradients via backpropagation."""
--> 288         return self._func(self, X, is_train=is_train)
    289 
    290     def initialize(self, X: Optional[InT] = None, Y: Optional[OutT] = None) -> "Model":

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/layers/chain.py in forward(model, X, is_train)
     52     callbacks = []
     53     for layer in model.layers:
---> 54         Y, inc_layer_grad = layer(X, is_train=is_train)
     55         callbacks.append(inc_layer_grad)
     56         X = Y

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/model.py in __call__(self, X, is_train)
    286         """Call the model's `forward` function, returning the output and a
    287         callback to compute the gradients via backpropagation."""
--> 288         return self._func(self, X, is_train=is_train)
    289 
    290     def initialize(self, X: Optional[InT] = None, Y: Optional[OutT] = None) -> "Model":

~/anaconda3/envs/acl/lib/python3.8/site-packages/spacy_transformers/layers/trfs2arrays.py in forward(model, trf_datas, is_train)
     26             src = model.ops.reshape2f(trf_data.tensors[t_i], -1, trf_data.width)
     27             dst, get_d_src = apply_alignment(model.ops, trf_data.align, src)
---> 28             output, get_d_dst = pooling(dst, is_train)
     29             outputs.append(output)
     30             backprops.append((get_d_dst, get_d_src))

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/model.py in __call__(self, X, is_train)
    286         """Call the model's `forward` function, returning the output and a
    287         callback to compute the gradients via backpropagation."""
--> 288         return self._func(self, X, is_train=is_train)
    289 
    290     def initialize(self, X: Optional[InT] = None, Y: Optional[OutT] = None) -> "Model":

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/layers/reduce_mean.py in forward(model, Xr, is_train)
     16 
     17 def forward(model: Model[InT, OutT], Xr: InT, is_train: bool) -> Tuple[OutT, Callable]:
---> 18     Y = model.ops.reduce_mean(cast(Floats2d, Xr.data), Xr.lengths)
     19     lengths = Xr.lengths
     20 

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/backends/numpy_ops.pyx in thinc.backends.numpy_ops.NumpyOps.reduce_mean()

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/backends/numpy_ops.cpython-38-x86_64-linux-gnu.so in View.MemoryView.memoryview_cwrapper()

~/anaconda3/envs/acl/lib/python3.8/site-packages/thinc/backends/numpy_ops.cpython-38-x86_64-linux-gnu.so in View.MemoryView.memoryview.__cinit__()

TypeError: a bytes-like object is required, not 'cupy.core.core.ndarray'
```

When I try en_core_web_sm (3.0.0) and  en_core_web_lg (3.0.0), there is no problem at all. 


## Your Environment
<!-- Include details of your environment. If you're using spaCy 1.7+, you can also type `python -m spacy info --markdown` and copy-paste the result here.-->
## Info about spaCy

- **spaCy version:** 3.0.1
- **Platform:** Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.10
- **Python version:** 3.8.5
- **Pipelines:** en_core_web_trf (3.0.0), en_core_web_sm (3.0.0), en_core_web_lg (3.0.0)

* Operating System: Win10, WSL2
* Python Version Used: Python 3.8.5
* spaCy Version Used: 3.0.1
* Environment Information: Virtual Environment

Hints from issue discussion:
Hi, can you check that you also have the latest version of `spacy-transformers` (`pip install -U spacy-transformers`)?
> Hi, can you check that you also have the latest version of `spacy-transformers` (`pip install -U spacy-transformers`)?

Yes,  I do have the latest version of 'spacy-transformers'.  I did `pip install -U spacy-transformers` and the same errors returned. 

Thanks.
Hmm, I'm not really sure what's going on. We saw a similar error recently was when `spacy.require_gpu()` was called after the model was loaded. Can you double-check that you are running `require_gpu()` before loading the model? Especially if things might have been executed out-of-order in a notebook, maybe this is a possible source of the error?

If that doesn't help, this is a bit frustrating as a suggestion, but my next suggestion would be to try again in a new venv because pip doesn't always clean up older files when upgrading packages, and it's possible something wasn't upgraded cleanly.
I'm getting this same error on Windows 10 when trying pass a sentence for a self-trained Transformer and NER pipeline for Finnish. This is happening on a fresh Conda environment (Python 3.8.5) with the latest versions. Errors relating to CuPy have appeared erratically for me. On another environment, I was unable run training on GPU, but that seems to function on this fresh installation. The error described above persists, however. I also made sure to call `require_gpu()` before loading the model.
> Hmm, I'm not really sure what's going on. We saw a similar error recently was when `spacy.require_gpu()` was called after the model was loaded. Can you double-check that you are running `require_gpu()` before loading the model? Especially if things might have been executed out-of-order in a notebook, maybe this is a possible source of the error?
> 
> If that doesn't help, this is a bit frustrating as a suggestion, but my next suggestion would be to try again in a new venv because pip doesn't always clean up older files when upgrading packages, and it's possible something wasn't upgraded cleanly.

Hi, I double-checked my code and I'm sure that the require_gpu() was before loading the model.  I also started a new conda virtual environment and reinstalled spacy v3 following the instruction:
pip install -U pip setuptools wheel
pip install -U spacy[cuda110]
python -m spacy download en_core_web_trf

and I got the same error.

However, I noticed the following:
1. If I don't use spacy.prefer_gpu() or require_gpu(0), there is no problem with doc = nlp(sent); It means that we are no problem with running spacy-transformer using cpu, right?
2. If I use spacy.prefer_gpu() or require_gpu(0) and load the "en_core_web_sm" or "en_core_web_lg" models instead of "en_core_web_trf", there is no problem as well. So there is no problem with  "en_core_web_sm" or "en_core_web_lg" using gpu, but "en_core_web_trf".
3. So, I'm wondering if the latest version of Spacy compatible with cuda-11 or not?

I appreciate your help.

I can't test any of this locally so this is a bit tedious, sorry. CUDA 11.0 support shouldn't be the issue. If `en_core_web_lg` is working fine with GPU, then spacy and cupy are probably fine. Someone else is reporting the same error in #6971 with a conda install of torch (the next thing I was going to suggest), so I am not sure what's going on and I can't replicate this with a very similar setup with conda in linux, so it's hard to debug.

What is the output of `conda list`?

Is it possible for you try this in a pip-only venv?
Hi all. As @adrianeboyd mentioned, I've posted the same problem in #6971. To keep the discussion in one thread, I'll post things here.

In my setup, I've used both `conda install` and `pip install` inside a `conda` environment. So I've tried using pip-only venv as suggested, but the same error still shows up.
@ilijaplusx: Is it windows or linux?
Amazon Linux 2, based on RHEL
What does `conda list` show?

Commit message: Initial warning

Patch to review:
diff --git a/spacy/errors.py b/spacy/errors.py
index 79a15fbaa41..b8de8f5b3cf 100644
--- a/spacy/errors.py
+++ b/spacy/errors.py
@@ -147,6 +147,11 @@ class Warnings:
             "will be included in the results. For better results, token "
             "patterns should return matches that are each exactly one token "
             "long.")
+    W111 = ("Jupyter notebook detected: if using `prefer_gpu()` or "
+            "`require_gpu()`, include it in the same cell right before "
+            "`spacy.load()` to ensure that the model is loaded on the correct "
+            "device. More information: "
+            "http://spacy.io/usage/v3#jupyter-notebook-gpu")
 
 
 @add_codes
diff --git a/spacy/language.py b/spacy/language.py
index 5bd1545d249..d98e827d6a6 100644
--- a/spacy/language.py
+++ b/spacy/language.py
@@ -20,7 +20,7 @@
 from .training import Example, validate_examples
 from .training.initialize import init_vocab, init_tok2vec
 from .scorer import Scorer
-from .util import registry, SimpleFrozenList, _pipe, raise_error
+from .util import registry, SimpleFrozenList, _pipe, raise_error, is_in_jupyter
 from .util import SimpleFrozenDict, combine_score_weights, CONFIG_SECTION_ORDER
 from .lang.tokenizer_exceptions import URL_MATCH, BASE_EXCEPTIONS
 from .lang.punctuation import TOKENIZER_PREFIXES, TOKENIZER_SUFFIXES
@@ -1615,6 +1615,11 @@ def from_config(
                 or lang_cls is not cls
             ):
                 raise ValueError(Errors.E943.format(value=type(lang_cls)))
+        # Warn about require_gpu if a jupyter notebook + cupy is detected
+        if is_in_jupyter():
+            from thinc.backends.cupy_ops import CupyOps
+            if CupyOps.xp is not None:
+                warnings.warn(Warnings.W111)
         # Note that we don't load vectors here, instead they get loaded explicitly
         # inside stuff like the spacy train function. If we loaded them here,
         # then we would load them twice at runtime: once when we make from config,
diff --git a/website/docs/api/top-level.md b/website/docs/api/top-level.md
index 37f619f3e3c..cb5843169e0 100644
--- a/website/docs/api/top-level.md
+++ b/website/docs/api/top-level.md
@@ -138,6 +138,13 @@ data has already been allocated on CPU, it will not be moved. Ideally, this
 function should be called right after importing spaCy and _before_ loading any
 pipelines.
 
+<Infobox variant="warning" title="Jupyter notebook usage">
+
+In a Jupyter notebook, run `prefer_gpu()` in the same cell as `spacy.load()`
+to ensure that the model is loaded on the correct device.
+
+</Infobox>
+
 > #### Example
 >
 > ```python
@@ -158,6 +165,13 @@ if no GPU is available. If data has already been allocated on CPU, it will not
 be moved. Ideally, this function should be called right after importing spaCy
 and _before_ loading any pipelines.
 
+<Infobox variant="warning" title="Jupyter notebook usage">
+
+In a Jupyter notebook, run `require_gpu()` in the same cell as `spacy.load()`
+to ensure that the model is loaded on the correct device.
+
+</Infobox>
+
 > #### Example
 >
 > ```python
@@ -177,6 +191,13 @@ Allocate data and perform operations on CPU. If data has already been allocated
 on GPU, it will not be moved. Ideally, this function should be called right
 after importing spaCy and _before_ loading any pipelines.
 
+<Infobox variant="warning" title="Jupyter notebook usage">
+
+In a Jupyter notebook, run `require_cpu()` in the same cell as `spacy.load()`
+to ensure that the model is loaded on the correct device.
+
+</Infobox>
+
 > #### Example
 >
 > ```python
diff --git a/website/docs/usage/v3.md b/website/docs/usage/v3.md
index 79a3225e00b..1329d2c293b 100644
--- a/website/docs/usage/v3.md
+++ b/website/docs/usage/v3.md
@@ -1168,3 +1168,15 @@ This means that spaCy knows how to initialize `my_component`, even if your
 package isn't imported.
 
 </Infobox>
+
+#### Using GPUs in Jupyter notebooks {#jupyter-notebook-gpu}
+
+In Jupyter notebooks, run [`prefer_gpu`](/api/top-level#spacy.prefer_gpu),
+[`require_gpu`](/api/top-level#spacy.require_gpu) or
+[`require_cpu`](/api/top-level#spacy.require_cpu) in the same cell as
+[`spacy.load`](/api/top-level#spacy.load) to ensure that the model is loaded on the correct device.
+
+Due to a bug related to `contextvars` (see the [bug
+report](https://github.com/ipython/ipython/issues/11565)), the GPU settings may
+not be preserved correctly across cells, resulting in models being loaded on
+on the wrong device or only partially on GPU.


Code review comments:
diff_hunk: @@ -1615,6 +1615,11 @@ def from_config(
                 or lang_cls is not cls
             ):
                 raise ValueError(Errors.E943.format(value=type(lang_cls)))
+        # Warn about require_gpu if a jupyter notebook + cupy is detected
+        if is_in_jupyter():
+            from thinc.backends.cupy_ops import CupyOps
+            if CupyOps.xp is not None:
+                warnings.warn(Warnings.W111)
path: spacy/language.py
text: @user1:
I don't really love the idea of just displaying this warning always, no matter what the user does. We should try to only warn if the user gets it wrong.

How about this:

* We set a global variable reflecting that the GPU has been enabled in some context. This global isn't going to work with threads or contexts, it's not the "source of truth". But it does tell us that something was done.
* When we load, we check whether the following is true: a) we're in a Jupyter notebook, b) The "use gpu" global is set, c) The "use gpu" context var isn't set. If all of those are true we should know we have a problem and can warn, right?

@author:
Should we move all this logic into thinc, or do you just want to handle common spacy misusage here?