Repository: kedro-org/kedro
Pull request: [DOC] dynamic session loading in non jupyter environments (marimo, python scripts, streamlit)
## Description

As discussed in #4440 currently kedro users that decide to not use jupyter for exploration cannot dynamically load their kedro config (catalog, credentials, etc) as those non jupyter environments lack the `%` magic commands that kedro currently relies on for its ipython/jupyter integration

This PR simply documents how to go ahead with such integration with today's codebase, making use of the private `kedro.utils._find_kedro_project`
## Development notes

nothing changed in development, just a doc PR

## Developer Certificate of Origin
We need all contributions to comply with the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). All commits must be signed off by including a `Signed-off-by` line in the commit message. [See our wiki for guidance](https://github.com/kedro-org/kedro/wiki/Guidelines-for-contributing-developers/).

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

## Checklist

- [x] Read the [contributing](https://github.com/kedro-org/kedro/blob/main/CONTRIBUTING.md) guidelines
- [x] Signed off each commit with a [Developer Certificate of Origin (DCO)](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/managing-the-commit-signoff-policy-for-your-repository)
- [x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
- [x] Updated the documentation to reflect the code changes
- [x] Added a description of this change in the [`RELEASE.md`](https://github.com/kedro-org/kedro/blob/main/RELEASE.md) file
- [x] Added tests to cover my changes
- [x] Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team


Problem statement:
`kedro marimo` + better programmatic setup for non jupyter/ipython envs
## Description
I'm frustrated by Kedro's heavy reliance on IPython magics (%reload_kedro) for notebook setup. While this works for Jupyter, it creates barriers for:
- Modern notebook interfaces like marimo that don't support IPython magics
- Script-based workflows where magics aren't available, though `kedro run` is often used for those
- Preferring explicit setup over magic injections

I first encountered this issue when importing data from the catalog in a marimo notebook. If `kedro-project/` is the root of my project, my notebook sits in `kedro-project/nbs/notebook.py`. I find that if I run the notebook from `kedro-project/` or from `kedro-project/nbs/` the behaviour isn't the same. I currently use this code to load the config:

```python
from kedro.io import DataCatalog
from kedro.config import OmegaConfigLoader
from kedro.framework.project import settings
from pathlib import Path
conf_loader = OmegaConfigLoader(
    conf_source=Path(__file__).parent /settings.CONF_SOURCE,
    default_run_env = "base"
)

catalog = DataCatalog.from_config(conf_loader["catalog"], credentials=conf_loader["credentials"])
mytable = catalog.load("mytable")
```

from `kedro-project/nbs`:
```bash
marimo edit notebook.py
```
-> ❌ fails as it thinks the filepath for `mytable` is relative to the current directory  instead of the project's root

from `kedro-project`:
```bash
marimo edit nbs/notebook.py
```
-> ✅ works fine because we are in the project's root

my catalog looks something like:
```yaml
mytable:
  type: pandas.SQLQueryDataset
  credentials: postgres_dwh
  filepath: conf/base/sql/mytable.sql
```

## Context

To use kedro in marimo notebooks more easily. It has been suggested to parametrise the catalog with a file path but I don't think that's a very nice solution and hopefully this can be implemented automatically without magic outside of jupyter/ipython.

## Possible Implementation

I've dug around the `%reload_kedro` magic code and I think we could re use some of that, I haven't fully tested it but I think it should work e.g.:

```python
from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
from kedro.framework.project import configure_project
from kedro.utils import _find_kedro_project
from pathlib import Path

# Get project root
notebook_path = Path(__file__).resolve().parent
project_root = _find_kedro_project(notebook_path)

# Now create the session
with KedroSession.create(project_path=project_root) as session:
    context = session.load_context()
    catalog = context.catalog
    weekly_sales = catalog.load("mytable")
```

The point here is to make `_find_kedro_project` available in the public API and document this way of setting up a kedro session for non jupyter users.

## PS

- I could be wrong about some of the assumptions I have made, perhaps the issue is specific to `catalog.load` but I think `kedro` could benefit from a programmatic magic-free way of connecting to the project's config/catalog
- I connected with @mscolnick from the marimo team on perhaps introducing `kedro marimo` command, I think it would be a nice merging of 2 worlds.

Hints from issue discussion:
I read the thread in marimo but I cannot signup/login to Discord so I can only comment here.

The majority of the magic function is making sure the root path is set correctly. They are not too important in the context of using absolute path but as Python cares where do you actually execute the command from.

The function itself is rather simple, it's mostly a syntax sugar to get started quickly but I suppose we should have some documentation on how to do this programmatically. As long as the session is created in the correct path, it should work.

What exactly is failing from marimo? In any case I think it will be an easy fix. 
I read the thread in marimo but I cannot signup/login to Discord so I can only comment here.

The majority of the magic function is making sure the root path is set correctly. They are not too important in the context of using absolute path but as Python cares where do you actually execute the command from.

The function itself is rather simple, it's mostly a syntax sugar to get started quickly but I suppose we should have some documentation on how to do this programmatically. As long as the session is created in the correct path, it should work.

https://docs.kedro.org/en/stable/api/kedro.framework.session.session.KedroSession.html#kedro.framework.session.session.KedroSession

The main way to run kedro programmatically is to create a Kedro Session, this is what the magic command do behind the scene. The link provided an example, and basically you need to provide the correct path to the bootstrap function and you should be good to go. 
Agree with @noklam that you should be able to create your `KedroSession` even without the IPython magic: https://docs.kedro.org/en/stable/kedro_project_setup/session.html

The magic just makes it slightly more convenient.

> I connected with @mscolnick from the marimo team on perhaps introducing `kedro marimo` command, I think it would be a nice merging of 2 worlds.

This is much broader scoped, but IMO would be very valuable. I actually started looking into this last year, but it would be very helpful to try to figure out what we want to accomplish. In the past, we had `kedro jupyter convert` (eventually deprecated due to low use), but the goal was to move people away from Jupyter notebooks into more production-ready data pipelines.

Marimo is different (I ~like~**love** the `kedro marimo edit` phrasing) in that Marimo code is just Python. Furthermore, Marimo notebooks following best practices (wrapping cells in functions, clearly defining inputs and outputs) map well to Kedro nodes and pipelines.

Obviously needs some more thought, but I think the right way to think about this might actually be:

1. Programmatically convert a Kedro pipeline to a Marimo notebook (and view using the Marimo UI)

    There are some things that need to be figured out here, like what should map to a notebook (a single registered pipeline? a single modular pipeline?). I think a modular pipeline is a good start for a PoC.

    Also, I've always wondered how to best represent datasets. Does Marimo have a standardized way of representing external resources? Else this can just be Python function calls to read and write the data.

    Finally, does it have to be a `.py` file, or is there some in-memory representation (the AST)? One-way conversion is a very poor user experience, which brings us to...
2. Figure out how to write back modifications to the Kedro project

A `kedro marimo export` could be a nice (easy) piece of functionality to add on, but this would honestly be the least useful thing to have.

@lucharo would also love to hear if this vision for a `kedro marimo` command—essentially Marimo as an editor for pipelines—is aligned with what would be useful to you, and what the Marimo team (@mscolnick @akshayka) are thinking/would be excited about.

**Disclaimer**: This is very much stream-of-consciousness and based on what I remember of looking into doing this in the past; may edit later.
Thanks a lot @lucharo for opening this issue!

I want to log that ultimately this connected to Kedro's magic handling of relative paths in `catalog.yml`, first reported on https://github.com/kedro-org/kedro/issues/1934, discussed at length on a Tech Design session that didn't reach any substantive consensus https://github.com/kedro-org/kedro/issues/2924#issuecomment-1690280993, and also in https://github.com/kedro-org/kedro/issues/2965 

Your solution using `_find_kedro_project` looks appropriate 👍🏼 and maybe indeed the first step should be making it part of the public API.
> Does Marimo have a standardized way of representing external resources? Else this can just be Python function calls to read and write the data.

We don't currently, though it's an interesting idea. 

> Finally, does it have to be a .py file, or is there some in-memory representation (the AST)?

marimo does have an in-memory representation. There is no public API yet, but for the purposes of this integration we could experiment using internal APIs if needed

> Figure out how to write back modifications to the Kedro project ... if this vision for a kedro marimo command—essentially Marimo as an editor for pipelines—is aligned with what would be useful to you, and what the Marimo team (@mscolnick @akshayka) are thinking/would be excited about.

"marimo as an editor for pipelines" is an exciting vision indeed. @lucharo, curious if it resonates with you?



Hi all, apologies for the delayed response. I am glad that opening this issue has sparked enthusiasm from both the Kedro and Marimo teams. 

I see this issue breaking down into 3 sub-issues/potential Kedro-Marimo crossovers, let me explain:

1. better compatibility of Kedro catalog with general scripts as well as non-jupyter notebook environment (e.g. Marimo - I don't know any other tbh). This would include making the `_find_project` function part of the public API and adding a piece of documentation on how to locate the config/catalog automatically within a Kedro project (without using ipython magic). I will try out the KedroSession + `_find_project` logic and see if I find any issue @noklam 

2. what @deepyaman brings up. I am a big fan of moving Kedro users away from jupyter notebooks. Currently, in my team, pipelines are developed in Jupyter and then turned into kedro pipelines/scripts. It's a very sub-optimal process and the development to production gap is too long. I am a big fan of kedro-only workflows (dev + production all in one). 

Kedro and Marimo are already similar enough for the symbiosis that @deepyaman suggests to happen, you both define DAGs and isolate nodes within functions. @deepyaman suggests turning kedro into marimo notebooks and I would like to suggest doing the opposite. Support kedro as an output format for marimo: `marimo export kedro pipeline.py`

Finally "marimo as an editor for pipelines" is indeed an exciting vision @akshayka, does it align with marimo's roadmap/mission? 
to close the first simpler thread of creating kedro sessions programmatically from marimo or python scripts, this is the code:

```py
from pathlib import Path
from kedro.utils import _find_kedro_project
from kedro.framework.startup import bootstrap_project
from kedro.framework.session import KedroSession

current_file_dir = Path(__file__).resolve().parent
project_root = _find_kedro_project(current_file_dir)
bootstrap_project(Path(project_root))

with KedroSession.create(project_path=project_root) as session:
    context = session.load_context()
    catalog = context.catalog
    parameters = catalog.load('parameters')
    table = catalog.load('table')
```

I'd like to make my first doc PR to kedro, where would be good @astrojuanlu @noklam @deepyaman??
@lucharo Excellent! Create a new doc under `docs/source/integrations/`, we'll take it from there!
@astrojuanlu do you think it should be under integrations or modifying https://docs.kedro.org/en/stable/kedro_project_setup/session.html?

The one concern I have is that this is using "private" functionality, and that means we're either limiting our ability to change what `_find_kedro_project` does (or risk breaking workflows for people who followed the docs). You suggested above a first step could be making it part of the public API--probably needs some discussion, but likely fine--in which case I don't think we need a separate integration doc.

Alternatively, we could view this is as the start of a Marimo integration page, @lucharo can showcase more in the way using Kedro with Marimo. And I think maybe for now, just mark this experimental/evolving, noting that it uses a currently-private API?
There's two sides to this:

- Users wanting to find "how to Kedro and marimo" = Good to have a standalone integration page
- How this should be the norm for starting Kedro sessions = Possible modifications to that session page as well as the public API

We can look at these separately. At least the marimo integration can be a driver for the latter.

Commit message: added item to release.md

Signed-off-by: lucharo <luis@merqato.eu>

Patch to review:
diff --git a/RELEASE.md b/RELEASE.md
index e974dd5bcc..348465111e 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -6,6 +6,7 @@
 
 ## Bug fixes and other changes
 * Updated `_LazyDataset` representation when printing `KedroDataCatalog`.
+* Added non-jupyter environment integration page (e.g. marimo) with dynamic kedro session loading.
 
 ## Breaking changes to the API
 ## Documentation changes
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 5575a8607b..6e581b8161 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -88,6 +88,7 @@ Welcome to Kedro's award-winning documentation!
 
    integrations/pyspark_integration.md
    integrations/mlflow.md
+   integrations/marimo.md
 
 .. toctree::
    :maxdepth: 2
diff --git a/docs/source/integrations/marimo.md b/docs/source/integrations/marimo.md
new file mode 100644
index 0000000000..649658f068
--- /dev/null
+++ b/docs/source/integrations/marimo.md
@@ -0,0 +1,36 @@
+# How to use Kedro from Marimo and non-Jupyter environments
+
+This guide explains how to set up Kedro programmatically without relying on IPython magics, making it compatible with non-Jupyter environments (python scripts or streamlit apps) and modern notebook interfaces like Marimo.
+
+## Overview
+
+Kedro Jupyter notebook users rely on the magic command `%load_ext kedro.ipython` to source their catalog, context, pipelines and session objects, see [Kedro and Notebooks](../notebooks_and_ipython/kedro_and_notebooks.md) for more. This magic command automatically finds the location of the root kedro project, avoiding having users hardcode paths to their configuration files.
+
+Such magic commands are not available in modern notebook interfaces such as marimo or in simple python scripts. Yet, using Kedro for sharing configuration, credentials or the data catalog across a project is still desired when developing data science projects.
+
+## Dynamically loading kedro config
+
+Instead of relying on magic, use the following script to automatically find the kedro root and load the kedro session objects:
+
+```python
+from pathlib import Path
+
+from kedro.framework.session import KedroSession
+from kedro.framework.startup import bootstrap_project
+from kedro.framework.project import configure_project
+from kedro.utils import _find_kedro_project
+
+# Get project root
+current_dir = Path(__file__).resolve().parent
+project_root = _find_kedro_project(current_dir)
+bootstrap_project(Path(project_root))
+
+# Create and use the session
+with KedroSession.create(project_path=project_root) as session:
+    context = session.load_context()
+    catalog = context.catalog
+    credentials = context.config_loader["credentials"]
+    data = catalog.load("mytable")
+```
+
+This enables loading Kedro settings, including catalog and credentials, without using IPython magic.


Code review comments:
diff_hunk: @@ -0,0 +1,36 @@
+# How to use Kedro from Marimo and non-Jupyter environments
+
+This guide explains how to set up Kedro programmatically without relying on IPython magics, making it compatible with non-Jupyter environments (python scripts or streamlit apps) and modern notebook interfaces like Marimo.
+
+## Overview
+
+Kedro Jupyter notebook users rely on the magic command `%load_ext kedro.ipython` to source their catalog, context, pipelines and session objects, see [Kedro and Notebooks](../notebooks_and_ipython/kedro_and_notebooks.md) for more. This magic command automatically finds the location of the root kedro project, avoiding having users hardcode paths to their configuration files.
+
+Such magic commands are not available in modern notebook interfaces such as marimo or in simple python scripts. Yet, using Kedro for sharing configuration, credentials or the data catalog across a project is still desired when developing data science projects.
path: docs/source/integrations/marimo.md
text: @user1:
While factually correct, I think I'd focus this page on describing how to run Kedro on marimo, so trim down this introduction a bit. Some other possible additions:

- Do not assume that the reader knows marimo, hence include a sentence linking to its official website and explaining briefly what it is
- Add the necessary `marimo edit` or `marimo run` commands
- Add a screenshot that demonstrates how this looks like

@author:
Would you want to focus solely on marimo? I think it could be relevant to talk about non jupyter environments as well: streamlit, python scripts or even cli scripts

@author:
and also, what is the screenshot meant to show? I think what I might do is create a start that uses marimo, I'd need a screenshot of a starter project that's not my work projects anyway