Repository: kedro-org/kedro

Issue:
%load_node does not work with notebook>=7.2.0
## Description
Based on the [latest guide on `main` branch](https://github.com/kedro-org/kedro/blob/b1b7f13a58b86b94390bd7bcc7db71f761e8c48b/docs/source/notebooks_and_ipython/kedro_and_notebooks.md) I've tried running notebook with Kedro kernel. While most of the functionality seems OK, the `%load_node` feature does not seem to work correctly.

## Context
Not being able to use `%load_node` in Jupyter Notebook with Kedro loaded.

## Steps to Reproduce
1. Follow instructions from [latest guide on `main` branch](https://github.com/kedro-org/kedro/blob/b1b7f13a58b86b94390bd7bcc7db71f761e8c48b/docs/source/notebooks_and_ipython/kedro_and_notebooks.md) 
3. Run `kedro jupyter notebook`
4. In new notebook, run `%load_node split_data_node`

## Expected Result
Function works properly as with `notebook==7.1.2`.
<img width="1205" alt="Screenshot 2024-12-05 at 22 28 34" src="https://github.com/user-attachments/assets/385c52a4-94bb-4811-ae22-8e95ab43530a">


## Actual Result
<img width="1205" alt="Screenshot 2024-12-05 at 22 30 17" src="https://github.com/user-attachments/assets/aea37393-29b7-43f8-9bb5-79a84d0e2098">

## Additional info
Works fine when version of `notebook` is pinned to `7.1.2`. I did not do bisection between `7.1.2` and `7.3.1` to find where it broke though.


## Your Environment

* Kedro version used (`pip show kedro` or `kedro -V`): `0.19.10`
* Python version used (`python -V`): `Python 3.10.10`
* Operating system and version: macOS 14.3.1 (23D60) (ARM)



Discussion:
Thank you for reporting this @marrrcin ! We'll have somebody else on the team try to reproduce this.
Thanks, @marrrcin. I confirm that starting from Notebook 7.2.0, this feature stopped working. It creates notebook cells, but they are empty instead of containing the node code.

After some digging, it appears that a recent change to Jupyter Notebook (version 7.2 and above) introduced a feature called “Full notebook windowing mode” which is now enabled by default:

[Changelog reference](https://jupyter-notebook.readthedocs.io/en/stable/changelog.html#full-notebook-windowing-mode-by-default)

This mode virtualises how cells are rendered, so only visible cells are injected into the DOM. As a result, any extension or magic command that relies on inserting new cells (e.g. via DOM manipulation or ipylab commands) may fail silently or have broken behavior.

In our case, the `%load_node` command from Kedro depends on ipylab commands, which used to work fine in older Notebook versions but now no longer responds in Notebook 7.2+ when “Full windowing mode” is active.

**Workaround / Solution:**

We can actually revert to the older behaviour through settings on jupyter notebook, change the “Windowing mode” setting from “full” to “defer.”:

1. Open your Notebook.
2. Go to Settings → Settings Editor → Notebook in the menu.
3. Scroll to “Windowing mode” and choose defer.

<img width="906" alt="Image" src="https://github.com/user-attachments/assets/95268547-3db3-4512-971b-039725335dbd" />

This should allows the `%load_node` command to work again as it did previously.

@DimedS @astrojuanlu @merelcht should we look into how to set this globally or programmatically or maybe just update our docs?


Ugh, that's really unfortunate... Thanks for digging @SajidAlamQB , this is helpful!

IMHO, if it doesn't work with the default settings it's unfortunately broken. Can we maybe rework the feature so that instead of inserting new cells, it writes everything to the current one?
> Ugh, that's really unfortunate... Thanks for digging [@SajidAlamQB](https://github.com/SajidAlamQB) , this is helpful!
> 
> IMHO, if it doesn't work with the default settings it's unfortunately broken. Can we maybe rework the feature so that instead of inserting new cells, it writes everything to the current one?

I like that proposal. It might also be possible to switch off the `Full Notebook Windowing` mode programmatically when running the %load_node command or modify the API we use to allow creating a new cell even with that mode enabled.
> It might also be possible to switch off the `Full Notebook Windowing` mode programmatically when running the %load_node command.

This solution could work, we would need to temporarily change the windowing mode to "defer" before running the cell creation commands, then restore it afterward. It will maintain compatibility with all notebook versions and doesn't require users to manually change settings and keeps the default option albeit a little bit hacky.


> This solution could work, we would need to temporarily change the windowing mode to "defer" before running the cell creation commands, then restore it afterward. It will maintain compatibility with all notebook versions and doesn't require users to manually change settings and keeps the default option albeit a little bit hacky.

looks like easiest solution if it will work
> Can we maybe rework the feature so that instead of inserting new cells, it writes everything to the current one?

@astrojuanlu @DimedS, I've tried playing around with switching the windowing mode programatically, but its a lot more complex than I anticipated I think @astrojuanlu initial suggestion of inserting it all into one is easiest. I have a PR that checks if the user is running `Jupyter Notebook >=7.2.0` and runs the output into 1 cell otherwise it uses the original approach. Let me know your thoughts.
Thanks for investigating! I'd even go as far as saying that we make this consistent for all `notebook` versions, to make it even simpler for us.



Patch:
diff --git a/RELEASE.md b/RELEASE.md
index c7804685ac..7d18ce5bbd 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -9,6 +9,7 @@
 * Fixed `MemoryDataset` to infer `assign` copy mode for Ibis Tables, which previously would be inferred as `deepcopy`.
 * Fixed pipeline packaging issue by ensuring `pipelines/__init__.py` exists when creating new pipelines.
 * Changed the execution of `SequentialRunner` to not use an executor pool to ensure it's single threaded.
+* Fixed `%load_node` magic command to work with Jupyter Notebook `>=7.2.0`.
 
 ## Breaking changes to the API
 ## Documentation changes
diff --git a/kedro/ipython/__init__.py b/kedro/ipython/__init__.py
index 20578a3e7c..376fdbdd7f 100644
--- a/kedro/ipython/__init__.py
+++ b/kedro/ipython/__init__.py
@@ -234,22 +234,19 @@ def magic_load_node(args: str) -> None:
     definition and a function call. If generating code is not possible, it will print
     the code instead.
     """
-
     parameters = parse_argstring(magic_load_node, args)
     node_name = parameters.node
 
     cells = _load_node(node_name, pipelines)
 
     run_environment = _guess_run_environment()
-    if run_environment == "jupyter":
-        # Only create cells if it is jupyter
-        for cell in cells:
-            _create_cell_with_text(cell, is_jupyter=True)
-    elif run_environment in ("ipython", "vscode"):
-        # Combine multiple cells into one
+
+    if run_environment in ("ipython", "vscode", "jupyter"):
+        # Combine multiple cells into one for IPython or VSCode or Jupyter
         combined_cell = "\n\n".join(cells)
-        _create_cell_with_text(combined_cell, is_jupyter=False)
+        _create_cell_with_text(combined_cell)
     else:
+        # For other environments or if detection fails, just print the cells
         _print_cells(cells)
 
 
@@ -283,17 +280,9 @@ def _find_var_positional_arg(self) -> str | None:
         return None
 
 
-def _create_cell_with_text(text: str, is_jupyter: bool = True) -> None:
-    if is_jupyter:
-        from ipylab import JupyterFrontEnd
-
-        app = JupyterFrontEnd()
-        # Noted this only works with Notebook >7.0 or Jupyter Lab. It doesn't work with
-        # VS Code Notebook due to imcompatible backends.
-        app.commands.execute("notebook:insert-cell-below")
-        app.commands.execute("notebook:replace-selection", {"text": text})
-    else:
-        get_ipython().set_next_input(text)  # type: ignore[no-untyped-call]
+def _create_cell_with_text(text: str) -> None:
+    """Create a new cell with the provided text content."""
+    get_ipython().set_next_input(text)  # type: ignore[no-untyped-call]
 
 
 def _print_cells(cells: list[str]) -> None:


Test patch:
diff --git a/tests/ipython/test_ipython.py b/tests/ipython/test_ipython.py
index 8cc6089fd1..30aac96c27 100644
--- a/tests/ipython/test_ipython.py
+++ b/tests/ipython/test_ipython.py
@@ -470,7 +470,7 @@ def test_load_node_with_jupyter(self, mocker, ipython):
 
         load_ipython_extension(ipython)
         ipython.magic("load_node dummy_node")
-        calls = [call("cell1", is_jupyter=True), call("cell2", is_jupyter=True)]
+        calls = [call("cell1\n\ncell2")]
         spy.assert_has_calls(calls)
 
     @pytest.mark.parametrize("run_env", ["ipython", "vscode"])