Repository: jupyterhub/jupyterhub

Issue:
'_xsrf' argument missing from POST (jupyterhub 4.1.0, jupyter-server-proxy 4.1.2)
### Bug description

Upgrading from zero to jupyterhub chart version 3.2.1 to 3.3.2 Most things seem fine, but externally launching applications, namely:
https://github.com/jupyterhub/jupyter-rsession-proxy
https://github.com/innovationOUtside/nb_serverproxy_openrefine
Do not open, giving an error in the browser (for RStudio) `Status code 403 returned by RStudio Server when executing 'client_init'` and in the singleuser container logs:
```
[E 2024-03-21 19:39:48.509 ServerApp] Not accepting cookie auth on POST /user/VRook%20%28WMF%29/rstudio/rpc/client_init: HTTP 403: Forbidden ('_xsrf' argument missing from POST)
[W 2024-03-21 19:39:48.510 ServerApp] 403 POST /user/VRook%20%28WMF%29/rstudio/rpc/client_init (10.100.1.0): '_xsrf' argument missing from POST
```

### How to reproduce

Install zero to jupyterhub chart version 3.3.2 with rstudio built into the singleuser container and launch rstudio. 

#### Expected behaviour

RStudio/OpenRefine open

#### Actual behaviour

RStudio/OpenRefine give an error

### Your personal set up

zero-to-jupyterhub chart 3.3.2 launching singleuser containers built out of ubuntu 22.04

https://github.com/toolforge/paws/pull/392


```
@PAWS:~$ pip freeze
aiohttp==3.9.3
aiosignal==1.3.1
alembic==1.13.1
altair==5.2.0
annotated-types==0.6.0
anyio==4.3.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-generator==1.10
async-lru==2.0.4
async-timeout==4.0.3
attrs==23.2.0
Babel==2.14.0
bash_kernel==0.9.3
beautifulsoup4==4.12.3
bleach==6.1.0
blinker==1.7.0
bokeh==3.3.4
bqplot==0.12.43
branca==0.7.1
cachetools==5.3.3
certifi==2024.2.2
certipy==0.1.3
cffi==1.16.0
charset-normalizer==3.3.2
cli_helpers==2.3.1
click==8.1.7
colorcet==3.1.0
comm==0.2.2
configobj==5.0.8
contourpy==1.2.0
cryptography==36.0.2
cycler==0.12.1
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
deltas==0.7.0
docopt==0.6.2
exceptiongroup==1.2.0
executing==2.0.1
fastapi==0.110.0
fastjsonschema==2.19.1
Flask==3.0.2
fonttools==4.50.0
fqdn==1.5.1
frozenlist==1.4.1
gitdb==4.0.11
GitPython==3.1.42
google==3.0.0
greenlet==3.0.3
h11==0.14.0
holoviews==1.18.3
httpcore==1.0.4
httpx==0.27.0
idna==3.6
ipydatawidgets==4.3.5
ipykernel==6.29.3
ipyleaflet==0.18.2
ipython==8.22.2
ipywidgets==8.1.2
isoduration==20.11.0
itsdangerous==2.1.2
jedi==0.19.1
Jinja2==3.1.3
joblib==1.3.2
json5==0.9.24
jsonable==0.3.1
jsonpointer==2.4
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.10.0
jupyter-lsp==2.2.4
jupyter-resource-usage==1.0.2
jupyter-rsession-proxy==2.2.1
jupyter-telemetry==0.1.0
jupyter_client==8.6.1
jupyter_core==5.7.2
jupyter_server==2.13.0
jupyter_server_proxy==4.1.2
jupyter_server_terminals==0.5.3
jupyterhub==4.1.0
jupyterlab==4.1.5
jupyterlab-link-share==0.3.0
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.4
jupyterlab_widgets==3.0.10
kiwisolver==1.4.5
labpawspublic @ git+https://github.com/toolforge/labpawspublic@08b8b4f89143055df024ebbbfb159b07f0f2a5b0
Lasagne==0.1
linkify-it-py==2.0.3
lxml==5.1.0
Mako==1.3.2
Markdown==3.6
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.8.3
matplotlib-inline==0.1.6
mdit-py-plugins==0.4.0
mdurl==0.1.2
memento-client==0.6.1
mistune==3.0.2
mpmath==1.3.0
multidict==6.0.5
mwapi==0.6.1
mwcli==0.0.3
mwdb==0.1.2
mwdiffs==0.0.2
mwoauth==0.4.0
mwparserfromhell==0.6.6
mwpersistence==0.2.4
mwreverts==0.1.5
mwsessions==0.0.2
mwtypes==0.3.2
mwxml==0.3.3
mycli==1.27.0
nb-serverproxy-openrefine @ git+https://github.com/innovationOUtside/nb_serverproxy_openrefine@828089e0397defe011e43ddd1e2ba067afc43b6a
nbclient==0.10.0
nbconvert==7.16.3
nbformat==5.10.3
nest-asyncio==1.6.0
nose==1.3.7
notebook==7.1.2
notebook_shim==0.2.4
numpy==1.26.4
oauthlib==3.2.2
overrides==7.7.0
packaging==23.2
pamela==1.1.0
pandas==2.2.1
pandocfilters==1.5.1
panel==1.3.8
para==0.0.8
param==2.0.2
parso==0.8.3
paws @ git+https://github.com/toolforge/ipynb-paws@147bbeb4dda35e94785814ca9a713c044229b25d
pexpect==4.9.0
pillow==10.2.0
platformdirs==4.2.0
plotly==5.20.0
pprofile==2.1.0
prometheus_client==0.20.0
prompt-toolkit==3.0.43
protobuf==4.25.3
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
py-heat==0.0.6
py-heat-magic==0.0.2
pyaes==1.6.1
pyarrow==15.0.2
pycparser==2.21
pydantic==2.6.4
pydantic_core==2.16.3
pydeck==0.8.1b0
pydot==2.0.0
Pygments==2.17.2
PyJWT==2.8.0
PyMySQL==1.1.0
pyOpenSSL==24.1.0
pyparsing==3.1.2
pyperclip==1.8.2
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
python-stdnum==1.20
python-wdqs @ git+https://github.com/yuvipanda/python-wdqs.git@272dc435bdd1778561a486b42521a87d5e80c516
pythreejs==2.4.2
pytz==2024.1
pyviz_comms==3.0.1
pywikibot @ file:///srv/paws/pwb
PyYAML==6.0.1
pyzmq==25.1.2
qtconsole==5.5.1
QtPy==2.4.1
referencing==0.34.0
requests==2.31.0
requests-oauthlib==1.4.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==13.7.1
rise==5.7.1
rpds-py==0.18.0
rpy2==3.5.15
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
scikit-learn==1.4.1.post1
scikit-neuralnetwork==0.7
scipy==1.12.0
seaborn==0.13.2
Send2Trash==1.8.2
simpervisor==1.0.0
six==1.16.0
smmap==5.0.1
sniffio==1.3.1
soupsieve==2.5
SQLAlchemy==2.0.28
sqlglot==23.0.5
sqlparse==0.4.4
sseclient==0.0.22
stack-data==0.6.3
starlette==0.36.3
stopit==1.1.2
streamlit==1.32.2
sympy==1.12
tabulate==0.9.0
tenacity==8.2.3
terminado==0.18.1
Theano==1.0.5
threadpoolctl==3.4.0
tinycss2==1.2.1
toml==0.10.2
tomli==2.0.1
toolz==0.12.1
tornado==6.4
tqdm==4.66.2
traitlets==5.14.2
traittypes==0.2.1
types-python-dateutil==2.9.0.20240316
typing_extensions==4.10.0
tzdata==2024.1
tzlocal==5.2
uc-micro-py==1.0.3
uri-template==1.3.0
urllib3==2.2.1
watchdog==4.0.0
wcwidth==0.2.13
webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
Werkzeug==3.0.1
widgetsnbextension==4.0.10
wmpaws==1.3
xyzservices==2023.10.1
yamlconf==0.2.3
yarl==1.9.4
```




Discussion:
@minrk this would be a jupyter-rsession-proxy issue, on being compatible with 4.1.0 -  whats your take?
> @minrk this would be a jupyter-rsession-proxy issue, on being compatible with 4.1.0 - whats your take?

I was kind of thinking that, though it happened with two (the only two external plugins that I have) at the same time. So it seemed a little more like it belonged here, though perhaps not.
Its a change in jupyterhub (software deployed by the helm chart, and no the helm chart project itself) that has caused this i think, but its security related breaking change rather than a bug i think. So the fix could be for the software interacting with jupyterhub to adjust to jupyterhub's new behavior rather than for jupyterhub to revert the change.
I've now tried to reproduce this, but failed - the image rocker/binder:4.3.3 launched to start `jupyterhub-singleuser` worked as intended, and it provides RStudio.

The issue will likely boil down to details on whats going on in the user environment, for example, are you using old versions of jupyterhub inside the user environment image or similar?
@consideRatio I do not believe it is JupyterHub 4.1 changes, but rather jupyter-server-proxy changes that would be responsible. I think you may see this with current jupyter-server-proxy and ~any Jupyter server.

I don't know how jupyter-rsession-proxy works or how you can influence the authentication of the requests it makes.
Both the old and the, attempted, upgraded instances are running jupyter_server_proxy 4.1.2 So that doesn't look like it changed. I went from jupyterhub 4.0.2 to 4.1.0 on both the singleuser container and the hub-k8s container.

I agree that it is likely something about my environment if RStudio is working in another deploy. Let me see if I can get an instance of rocker/binder:4.3.3 to see if it is working in the rest of my env.
ok, then it would probably be in jupyterhub 4.1, thanks for narrowing it down!

I've confirmed this with JupyterHub 4.1 and jupyter-server-proxy 4.1.2. I'm not sure this is a situation that should have changed, so I will investigate.

@consideRatio fwiw, rocker/binder:4.3.3 has jupyter-server-proxy 4.1.1 and jupyterhub 4.0.2, so isn't up to date on either package.
okay, I see and understand what's different. I will try to figure out if I can make a JupyterHub 4.1.1 that unbreaks this.

Resolving patch:
diff --git a/jupyterhub/services/auth.py b/jupyterhub/services/auth.py
--- a/jupyterhub/services/auth.py
+++ b/jupyterhub/services/auth.py
@@ -880,13 +880,39 @@ def _get_xsrf_token_id(self, handler):
 
     def _patch_xsrf(self, handler):
         """Patch handler to inject JuptyerHub xsrf token behavior"""
-        handler._xsrf_token_id = self._get_xsrf_token_id(handler)
-        # override xsrf_token property on class,
-        # so it's still a getter, not invoked immediately
-        handler.__class__.xsrf_token = property(
-            partial(get_xsrf_token, cookie_path=self.base_url)
-        )
-        handler.check_xsrf_cookie = partial(self.check_xsrf_cookie, handler)
+        if isinstance(handler, HubAuthenticated):
+            # doesn't need patch
+            return
+
+        # patch in our xsrf token handling
+        # overrides tornado and jupyter_server defaults,
+        # but not others.
+        # subclasses will still inherit our overridden behavior,
+        # but their overrides (if any) will take precedence over ours
+        # such as jupyter-server-proxy
+        for cls in handler.__class__.__mro__:
+            # search for the nearest parent class defined
+            # in one of the 'base' Handler-defining packages.
+            # In current implementations, this will
+            # generally be jupyter_server.base.handlers.JupyterHandler
+            # or tornado.web.RequestHandler,
+            # but doing it this way ensures consistent results
+            if (cls.__module__ or '').partition('.')[0] not in {
+                "jupyter_server",
+                "notebook",
+                "tornado",
+            }:
+                continue
+            # override check_xsrf_cookie where it's defined
+            if "check_xsrf_cookie" in cls.__dict__:
+                if "_get_xsrf_token_id" in cls.__dict__:
+                    # already patched
+                    return
+                cls._xsrf_token_id = property(self._get_xsrf_token_id)
+                cls.xsrf_token = property(
+                    partial(get_xsrf_token, cookie_path=self.base_url)
+                )
+                cls.check_xsrf_cookie = lambda handler: self.check_xsrf_cookie(handler)
 
     def check_xsrf_cookie(self, handler):
         """check_xsrf_cookie patch
@@ -931,8 +957,10 @@ async def _get_user_cookie(self, handler):
         token = self._get_token_cookie(handler)
         session_id = self.get_session_id(handler)
         if token and self._needs_check_xsrf(handler):
+            # call handler.check_xsrf_cookie instead of self.check_xsrf_cookie
+            # to allow subclass overrides
             try:
-                self.check_xsrf_cookie(handler)
+                handler.check_xsrf_cookie()
             except HTTPError as e:
                 self.log.error(
                     f"Not accepting cookie auth on {handler.request.method} {handler.request.path}: {e}"


Test patch: