Repository: holoviz/panel

Issue:
Performance  problems with panel > 0.13  and Lab3
We are experiencing some performance behaviors with panel > 0.13 
The problems we are facing are:
- General  lab slowness and input lags on Jupyter Lab 
- Random kernel crashes while using tools (can also happen in non Panel tools)
- On local Windows machined we cannot use any panel related imports in startup.py files, using panel in said files will cause the kernel to crash and never boot.


We get the following error:
```log
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/app.py", line 66, in main
    return SingleUserLabApp.launch_instance()
  File "/opt/conda/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 645, in start
    super().start()
  File "/opt/conda/lib/python3.9/site-packages/jupyter_server/serverapp.py", line 2669, in start
    self.start_ioloop()
  File "/opt/conda/lib/python3.9/site-packages/jupyter_server/serverapp.py", line 2655, in start_ioloop
    self.io_loop.start()
  File "/opt/conda/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 1875, in _run_once
    handle = self._ready.popleft()
IndexError: pop from an empty deque
```


Here is the boken system pip list:
```log
Package                       Version                     Editable project location
----------------------------- --------------------------- --------------------------------------------------------------------
aenum                         2.2.4
aiohttp                       3.8.1
aiosignal                     1.2.0
alabaster                     0.7.12
altair                        4.1.0
analytics-python              1.4.0
anyio                         3.6.1
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
asttokens                     2.0.5
async-timeout                 4.0.2
atomicwrites                  1.4.0
attrs                         21.4.0
awscli                        1.23.9
Babel                         2.10.1
backcall                      0.2.0
backoff                       1.10.0
beautifulsoup4                4.11.1
bleach                        5.0.0
bokeh                         2.4.3
boto3                         1.22.9
botocore                      1.25.9
cachetools                    5.2.0
certifi                       2022.5.18.1
cffi                          1.15.0
charset-normalizer            2.0.12
cloudpickle                   2.1.0
colorama                      0.4.4
coverage                      6.4.1
cycler                        0.11.0
debugpy                       1.6.0
decorator                     5.1.1
defusedxml                    0.7.1
docopt                        0.6.2
docutils                      0.15.2
entrypoints                   0.4
execnet                       1.9.0
executing                     0.8.3
fastjsonschema                2.15.3
filelock                      3.3.1
flake8                        3.8.3
fonttools                     4.33.3
frozenlist                    1.3.0
fsspec                        2022.5.0
gcsfs                         2022.5.0
google-api-core               2.8.1
google-auth                   2.6.6
google-auth-oauthlib          0.5.1
google-cloud-core             2.3.0
google-cloud-storage          2.3.0
google-crc32c                 1.3.0
google-resumable-media        2.3.3
googleapis-common-protos      1.56.2
hybridcontents                0.3.0
icdiff                        2.0.5
idna                          3.3
imagesize                     1.3.0
importlib-metadata            4.11.4
imubit-analytics              0.0.3
iniconfig                     1.1.1
iniherit                      0.3.9
ipyaggrid                     0.2.1
ipydatetime                   1.2.0
ipykernel                     6.13.1
ipython                       8.4.0
ipython-genutils              0.2.0
ipywidgets                    7.7.0
isort                         5.6.4
jedi                          0.18.1
jenkinsapi                    0.3.11
Jinja2                        3.1.2
jmespath                      1.0.0
joblib                        1.1.0
json5                         0.9.8
jsonschema                    4.6.0
jupyter                       1.0.0
jupyter-bokeh                 3.0.4
jupyter-client                7.3.3
jupyter-console               6.4.3
jupyter-core                  4.10.0
jupyter-server                1.17.1
jupyterlab                    3.4.2
jupyterlab-pygments           0.2.2
jupyterlab-server             2.14.0
jupyterlab-widgets            1.1.0
kiwisolver                    1.4.2
llvmlite                      0.38.1
Markdown                      3.3.7
MarkupSafe                    2.1.1
matplotlib                    3.5.1
matplotlib-inline             0.1.3
mccabe                        0.6.1
mistune                       0.8.4
mock                          3.0.5
mock-generator                1.0.0
monotonic                     1.6
multidict                     6.0.2
nbclassic                     0.3.7
nbclient                      0.6.4
nbconvert                     6.5.0
nbformat                      5.4.0
nbval                         0.9.6
nest-asyncio                  1.5.5
nose                          1.3.7
notebook                      6.4.12
notebook-shim                 0.1.0
numba                         0.55.2
numexpr                       2.8.1
numpy                         1.22.0
oauthlib                      3.2.0
packaging                     21.3
pandas                        1.2.5
pandocfilters                 1.5.0
panel                         0.13.1
param                         1.12.1
parso                         0.8.3
pickleshare                   0.7.5
Pillow                        9.1.1
pip                           22.1.2
pluggy                        0.13.1
pprintpp                      0.4.0
prometheus-client             0.14.1
prompt-toolkit                3.0.29
protobuf                      3.20.1
psutil                        5.9.1
psycopg2-binary               2.9.3
pure-eval                     0.2.2
py                            1.11.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycodestyle                   2.6.0
pycparser                     2.21
pyct                          0.4.8
pydantic                      1.9.0
pyflakes                      2.2.0
Pygments                      2.12.0
pymongo                       4.0.1
pyparsing                     3.0.9
PyPika                        0.48.8
pyrsistent                    0.18.1
pytest                        6.2.3
pytest-asyncio                0.14.0
pytest-cov                    2.11.1
pytest-forked                 1.4.0
pytest-icdiff                 0.5
pytest-mock                   3.6.1
pytest-repeat                 0.9.1
pytest-xdist                  2.4.0
python-dateutil               2.8.2
python-json-logger            0.1.11
python-slugify                6.1.2
pytimeparse                   1.1.8
pytz                          2022.1
pyviz-comms                   2.2.0
pywin32                       304
pywinpty                      2.0.5
PyYAML                        5.3.1
pyzmq                         23.1.0
qtconsole                     5.3.1
QtPy                          2.1.0
requests                      2.27.1
requests-oauthlib             1.3.1
rsa                           4.7.2
s3contents                    0.6.0
s3fs                          0.4.2
s3transfer                    0.5.2
scikit-learn                  1.1.1
scipy                         1.8.1
seaborn                       0.11.2
selenium                      3.141.0
Send2Trash                    1.8.0
setuptools                    62.3.3
shap                          0.40.0
simpleeval                    0.9.11
simplejson                    3.17.6
six                           1.16.0
sklearn                       0.0
slicer                        0.0.7
sniffio                       1.2.0
snowballstemmer               2.2.0
soupsieve                     2.3.2.post1
Sphinx                        3.2.1
sphinx-rtd-theme              0.5.0
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
stack-data                    0.2.0
tables                        3.7.0
tabulate                      0.8.9
terminado                     0.15.0
testpath                      0.6.0
text-unidecode                1.3
threadpoolctl                 3.1.0
timedb                        3.29.2
tinycss2                      1.1.1
toml                          0.10.2
toolz                         0.11.2
tornado                       6.1
traitlets                     5.2.2.post1
typing_extensions             4.2.0
urllib3                       1.26.9
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              1.3.2
wheel                         0.34.2
widgetsnbextension            3.6.0
yapf                          0.30.0
yarl                          1.7.2
yoyo-migrations               5.0.5
zipp                          3.8.0
```

And here is a pip list of the working environment:
```log
Package                           Version                     Editable project location
--------------------------------- --------------------------- ------------------------------------------------------------------
aenum                             2.2.4
aiohttp                           3.8.1
aiosignal                         1.2.0
alabaster                         0.7.12
altair                            4.1.0
analytics-python                  1.4.0
anyio                             3.6.1
argon2-cffi                       21.3.0
argon2-cffi-bindings              21.2.0
asttokens                         2.0.5
async-timeout                     4.0.2
atomicwrites                      1.4.0
attrs                             21.4.0
awscli                            1.23.9
Babel                             2.10.1
backcall                          0.2.0
backoff                           1.10.0
beautifulsoup4                    4.11.1
bleach                            5.0.0
bokeh                             2.4.3
boto3                             1.22.9
botocore                          1.25.9
cachetools                        5.2.0
certifi                           2022.5.18.1
cffi                              1.15.0
charset-normalizer                2.0.12
click                             8.0.3
cloudpickle                       2.1.0
colorama                          0.4.4
coverage                          6.4.1
cycler                            0.11.0
debugpy                           1.6.0
decorator                         5.1.1
defusedxml                        0.7.1
docopt                            0.6.2
docutils                          0.15.2
entrypoints                       0.4
execnet                           1.9.0
executing                         0.8.3
Faker                             13.3.4
fastjsonschema                    2.15.3
filelock                          3.3.1
flake8                            3.8.3
fonttools                         4.33.3
frozenlist                        1.3.0
fsspec                            2022.5.0
gcsfs                             2022.5.0
google-api-core                   2.8.1
google-auth                       2.7.0
google-auth-oauthlib              0.5.2
google-cloud-core                 2.3.1
google-cloud-storage              2.4.0
google-crc32c                     1.3.0
google-resumable-media            2.3.3
googleapis-common-protos          1.56.2
hybridcontents                    0.3.0
icdiff                            2.0.5
idna                              3.3
imagesize                         1.3.0
importlib-metadata                4.11.4
imubit-analytics                  0.0.3
iniconfig                         1.1.1
iniherit                          0.3.9
ipyaggrid                         0.2.1
ipydatetime                       1.2.0
ipykernel                         6.13.1
ipython                           8.4.0
ipython-genutils                  0.2.0
ipywidgets                        7.7.0
isort                             5.6.4
jedi                              0.18.1
jenkinsapi                        0.3.11
Jinja2                            3.1.2
jmespath                          1.0.0
joblib                            1.1.0
json5                             0.9.8
jsonschema                        4.6.0
jupyter-bokeh                     3.0.4
jupyter-client                    7.3.4
jupyter-contrib-core              0.3.3
jupyter-contrib-nbextensions      0.5.1
jupyter-core                      4.10.0
jupyter-highlight-selected-word   0.2.0
jupyter-latex-envs                1.4.6
jupyter-nbextensions-configurator 0.4.1
jupyter-server                    1.17.1
jupyterlab                        3.4.0
jupyterlab-pygments               0.2.2
jupyterlab-server                 2.14.0
jupyterlab-widgets                1.1.0
jupyterthemes                     0.20.0
kiwisolver                        1.4.3
kwargs                            1.0.1
lesscpy                           0.15.0
llvmlite                          0.38.1
lxml                              4.7.1
Markdown                          3.3.7
MarkupSafe                        2.1.1
matplotlib                        3.5.1
matplotlib-inline                 0.1.3
mccabe                            0.6.1
mistune                           0.8.4
mock                              3.0.5
mock-generator                    1.0.0
monotonic                         1.6
multidict                         6.0.2
mypy-extensions                   0.4.3
nbclassic                         0.3.7
nbclient                          0.6.4
nbconvert                         6.5.0
nbformat                          5.4.0
nbval                             0.9.6
nest-asyncio                      1.5.5
nose                              1.3.7
notebook                          6.4.12
notebook-shim                     0.1.0
numba                             0.55.2
numexpr                           2.8.1
numpy                             1.22.4
oauthlib                          3.2.0
packaging                         21.3
pandas                            1.2.5
pandocfilters                     1.5.0
panel                             0.13.1
param                             1.12.1
parameterized                     0.8.1
parso                             0.8.3
pathspec                          0.9.0
pbr                               5.8.1
pickleshare                       0.7.5
Pillow                            9.1.1
pip                               22.1.2
platformdirs                      2.4.1
pluggy                            0.13.1
ply                               3.11
pprintpp                          0.4.0
progress                          1.6
prometheus-client                 0.14.1
prompt-toolkit                    3.0.29
protobuf                          3.20.1
psutil                            5.9.1
psycopg2-binary                   2.9.3
pure-eval                         0.2.2
py                                1.11.0
pyasn1                            0.4.8
pyasn1-modules                    0.2.8
pycodestyle                       2.6.0
pycparser                         2.21
pyct                              0.4.8
pydantic                          1.9.0
pyflakes                          2.2.0
Pygments                          2.12.0
pymongo                           4.0.1
pyparsing                         3.0.9
PyPika                            0.48.8
pyrsistent                        0.18.1
PyRSS2Gen                         1.1
pytest                            6.2.3
pytest-asyncio                    0.14.0
pytest-cov                        2.11.1
pytest-datadir-ng                 1.1.1
pytest-forked                     1.4.0
pytest-icdiff                     0.5
pytest-mock                       3.6.1
pytest-pythonpath                 0.7.4
pytest-repeat                     0.9.1
pytest-xdist                      2.4.0
python-dateutil                   2.8.2
python-json-logger                0.1.11
python-slugify                    6.1.2
pytimeparse                       1.1.8
pytz                              2022.1
pyviz-comms                       2.2.0
pywin32                           304
pywinpty                          2.0.5
PyYAML                            6.0
pyzmq                             23.1.0
requests                          2.28.0
requests-oauthlib                 1.3.1
rsa                               4.7.2
s3contents                        0.6.0
s3fs                              0.4.2
s3transfer                        0.5.2
scikit-learn                      1.1.1
scipy                             1.8.1
seaborn                           0.11.2
selenium                          3.141.0
self                              2020.12.3
Send2Trash                        1.8.0
setuptools                        62.4.0
shap                              0.40.0
simpleeval                        0.9.11
simplejson                        3.17.6
six                               1.16.0
sklearn                           0.0
slicer                            0.0.7
sniffio                           1.2.0
snowballstemmer                   2.2.0
soupsieve                         2.3.2.post1
Sphinx                            3.2.1
sphinx-rtd-theme                  0.5.0
sphinxcontrib-applehelp           1.0.2
sphinxcontrib-devhelp             1.0.2
sphinxcontrib-htmlhelp            2.0.0
sphinxcontrib-jsmath              1.0.1
sphinxcontrib-qthelp              1.0.3
sphinxcontrib-serializinghtml     1.1.5
sqlparse                          0.4.2
stack-data                        0.2.0
success                           1.0.0
tables                            3.7.0
tabulate                          0.8.9
terminado                         0.15.0
testpath                          0.6.0
text-unidecode                    1.3
threadpoolctl                     3.1.0
timedb                            3.29.2
tinycss2                          1.1.1
toml                              0.10.2
tomli                             1.2.3
toolz                             0.11.2
tornado                           6.1
tqdm                              4.64.0
traitlets                         5.2.2.post1
typing_extensions                 4.2.0
urllib3                           1.26.9
versioneer                        0.22
virtualenv                        20.14.1
wcwidth                           0.2.5
webencodings                      0.5.1
websocket-client                  1.3.2
wheel                             0.34.2
widgetsnbextension                3.6.0
xeus-python                       0.13.6
xeus-python-shell                 0.2.0
yapf                              0.30.0
yarl                              1.7.2
yoyo-migrations                   5.0.5
zipp                              3.8.0
```



Discussion:
I was aware of performance regressions in 0.13.0 (see https://github.com/holoviz/panel/issues/3531) but was under the impression these were fixed. You are certain you have updated to 0.13.1 in your base env (the one that the Jupyter server is running in)?
Hey @philippjfr, I'm working with @ItamarShDev, we did tried 0.13.1 and even 0.14.0a1 and 0.14.0a2, but the problems listed above consisted.
Thanks for the update, someone will have to do a bisection on the codebase. Can you give me a very simple, step-wise recipe that will reproduce the issue so that bisection doesn't take forever?
The first two sections written above are hard to reproduce

The first one (the general slowness) is always there from the moment we starting Jupyter (with panel imported trough the startup folder). 

The second one (the kernel crashes) are happening suddenly during work, and the exception mentioned above is shown in the logs.

The third one (kernel wont start / crash) I can reproduce easily in windows, I'm creating a file with just line for importing panel and save it under the startup folder (`%user_home%\.ipython\profile_default\startup`).

We posted the list of packages above we are using above, we are using `jupyterlab` version `3.4.0` and we tried `panel` version `0.13.*` and above.

just to make it clearer when go back to older panel (0.12.7 and below) stuff are smoother, the kernel exception didn't happened during my test time and the import from windows using the startup folder worked smoothly and the kernel started without any issue. 

LMK if you need any other information.

thanks.
Thanks, that's very helpful!
Hey @philippjfr, do you have any news for us ?
If any tess are needed, we will be happy to test
Sorry, don't have any updates yet. Been a little swamped. Hope to have some time tomorrow.
Thanks for the response @philippjfr, looking forward for it.
I also experience this when working on my windows laptop on the current Panel master branch.
Sorry I haven't gotten to this. Unfortunately I've been travelling almost non stop for weeks and havent had access to a windows machine.
On this amazing youtube video by Thu Vue she has to state around minute 04:00 that you should not install Panel in the same virtual environment as Jupyterlab due to some weird issue https://www.youtube.com/watch?v=B11nK1QCPRA.

I believe this is most important bug to fix for Panel as you cannot currently work with the latest version of Panel and Jupyter lab on Windows if you are not an experienced user willing to put in some effort to get it working. 
Would be happy to backport this to a 0.13.x release asap if we figure out what is causing it. If someone with a windows machine could do a git bisect that would be tremendously helpful. I won't be able to do it until Sunday at the very earliest.
@philippjfr If you could guide me through, i can give you any Windows related info.
So sorry this has taken so long but I was finally able to test this on windows and I have had absolutely no luck reproducing the issue. Maybe we can sync at some point and debug together.
@maximlt or @Hoxbro do either of you have a windows machine where you could try to replicate?
By using the first environment on Windows and adding `import panel as pn` to a startup file, I get the same no response when running in Jupyter Lab. If I add another check to these lines `'jupyter_client' not in sys.modules`, it seems to work as expected.

https://github.com/holoviz/panel/blob/8c5ca7a1e16c84f781ee137d3f709fa2370e74d5/panel/io/server.py#L538-L545

I could not see any slowness except that. The log message in the first comment also seems to indicate it running on Linux and not Windows. 
Wow, great sleuthing thanks! Could you submit a PR adding that additional guard?

Resolving patch:
diff --git a/panel/io/server.py b/panel/io/server.py
--- a/panel/io/server.py
+++ b/panel/io/server.py
@@ -540,7 +540,9 @@ def create_static_handler(prefix, key, app):
     sys.version_info[:3] >= (3, 8, 0) and
     tornado.version_info >= (6, 1) and
     type(asyncio.get_event_loop_policy()) is asyncio.WindowsSelectorEventLoopPolicy and
-    'jupyter_server' not in sys.modules
+    (('jupyter_server' not in sys.modules and
+      'jupyter_client' not in sys.modules) or
+     'pytest' in sys.modules)
 ):
     asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
 


Test patch: