Repository: basilisp-lang/basilisp

Issue:
feature request: seamless import of Basilisp libraries in Python with implicit bootstrapping
Hi,

I'd like to request support for importing Basilisp libraries directly into Python applications like any standard Python library, without requiring explicit Basilisp bootstrapping or the user needing to know the library is written in Basilisp.

The goal is to enable writing Python packages in Basilisp that can be seamlessly imported into Python code, with Basilisp being implicitly bootstrapped as needed.

The [Bootstrapping](https://basilisp.readthedocs.io/en/latest/gettingstarted.html#bootstrapping) section of the Basilisp docs mentions `basilisp.main.bootstrap` and `basilisp.main.init` as bootstrapping methods, but it's unclear what happens when bootstrapping occurs multiple times, for example, if two Basilisp-based libraries each bootstrap themselves in their `__init__.py`.

The [basilisp.main](https://basilisp.readthedocs.io/en/latest/api/main.html) module also lists `basilisp.main.bootstrap_python`, but it's still unclear whether calling it multiple times is safe, or if the runtime will even be available without re-importing the module.

From reviewing the code, `basilisp.main.init` seems the most promising option

https://github.com/basilisp-lang/basilisp/blob/0fead055cf6435511c2b8e096de29276778be175/src/basilisp/main.py#L15-L30

Possibly in combination with a runtime check, such as whether `BasilispImporter` is already present in `sys.meta_path`, as done here:

https://github.com/basilisp-lang/basilisp/blob/0fead055cf6435511c2b8e096de29276778be175/src/basilisp/importer.py#L424-L432

Could you please advise of whether any of the existing bootstrapping methods can safely be used in this way (i.e., called multiple times at runtime) or suggest what changes might be needed to support this use case?

thanks

Discussion:
@chrisrink10: This is definitely something that I am interested in exploring. Do you have a more concrete example for what specifically you're looking to be able to do? Like how would you be executing this code?@ikappaki: I've published a simple example Basilisp library at https://github.com/ikappaki/basilex-python-lib to demonstrate this use case.

The package bootstraps Basilisp in its `__init__.py` using `basilisp.main.init`:

https://github.com/ikappaki/basilex-python-lib/blob/374844e060959600e4f954c667be942428f8bbe7/src/basilex_python_lib/__init__.py#L1-L3

It defines a Basilisp function in `basilex-python-lib.utils` namespace:

https://github.com/ikappaki/basilex-python-lib/blob/374844e060959600e4f954c667be942428f8bbe7/src/basilex_python_lib/utils.lpy#L1-L4

This function can be imported and used from a clean Python environment like any standard Python function:

```shell
$ python3 -m venv tmp1 
$ source tmp1/bin/activate
(tmp1)  $ pip install git+https://github.com/ikappaki/basilex-python-lib.git
Collecting git+https://github.com/ikappaki/basilex-python-lib.git
...

(tmp1)  $ python
Python 3.13.0 (v3.13.0:60403a5409f, Oct  7 2024, 00:37:40) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from basilex_python_lib import utils
>>> utils.hello("there")
:hello there
```

Note that in this pristine environment, with only the Basilisp-based package installed, there’s no need to manually bootstrap Basilisp or even be aware that the function was written in Basilisp. The library behaves just like any regular Python package.

The open question is whether using `basilisp.main.init` in `__init__.py` is the correct approach. If another Basilisp based library were imported the same way, it would call init again, yet the behavior of multiple calls isn't clearly documented or defined.

thanks@chrisrink10: That's helpful to clarify the question. I think using `basilisp.main.init` in the way you're using it is my recommended approach, but to your point I do believe that there is currently nothing guarding against repeated invocations across other libraries.

Those repeated invocations of `init()` would likely cause `basilisp.core` to be reloaded on each call, which seems like it could be problematic.

I think for this ticket, we could add some synchronization within `basilisp.main.init` to short-circuit and return early if the runtime is already initialized.

We could also modify the documentation to include this pattern in the `Bootstrapping` section of the `Getting Started` page.@ikappaki: 
> I think for this ticket, we could add some synchronization within `basilisp.main.init` to short-circuit and return early if the runtime is already initialized.
> 
> We could also modify the documentation to include this pattern in the `Bootstrapping` section of the `Getting Started` page.

Right, thanks! I'll stick with the `if any(isinstance(o, BasilispImporter) for o in sys.meta_path):  return` guard  as a workaround (mentioned earlier) while you consider the proper solution.

`__init__.py`
```python
import sys

from basilisp.main import init
from basilisp.importer import BasilispImporter

if any(isinstance(o, BasilispImporter) for o in sys.meta_path):
    print("pass")
else:    
    init()
```
@amano-kenji: Because python 3.13 has experimental free threading support, you might want to use a thread lock in `basilisp.main.init`.

Python is trying to [free itself from global interpreter lock](https://peps.python.org/pep-0703/).

Perhaps, basilisp can have `bootstrapped` atom that's thread safe? If you use a thread lock, an atom may be unnecessary.@amano-kenji: > We could also modify the documentation to include this pattern in the Bootstrapping section of the Getting Started page.

If the pull request documented this pattern, it would be perfect.

Solution patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1bbfa1b..f759d36 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,6 +18,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
  * `basilisp.core/str` now delegates to the builtin Python `str` in all cases except for customizing the string output for builtin Python types (#1237)
  * Optimised mainstream seq-consuming functions by coercing their inputs into `seq` upfront (#1234)
  * Renamed `awith` and `afor` to `with-async` and `for-async` for improved clarity (#1248)
+ * `basilisp.main.init` will only initialize the runtime environment on the first invocation (#1242)
 
 ### Fixed
  * Fix a bug where protocols with methods with leading hyphens in the could not be defined (#1230)
diff --git a/docs/gettingstarted.rst b/docs/gettingstarted.rst
index 4adb7d4..3ee9c87 100644
--- a/docs/gettingstarted.rst
+++ b/docs/gettingstarted.rst
@@ -120,8 +120,16 @@ Given a Basilisp entrypoint function ``main`` (taking no arguments) in the ``pro
 If you were to place this in a module such as ``myproject.main``, you could easily configure a `setuptools entry point <https://setuptools.pypa.io/en/latest/userguide/entry_point.html>`_ (or any analog with another build tool) to point to that script directly, effectively launching you directly to Basilisp code.
 
 For more sophisticated projects which may not have a direct or wrappable entrypoint, you can initialize the Basilisp runtime directly by calling :py:func:`basilisp.main.init` with no arguments.
-This may be a better fit for a project using something like Django, where the entrypoint is dictated by Django.
-In that case, you could use a hook such as Django's ``AppConfig.ready()``.
+A natural placement for this function call would be in the root ``__init__.py`` for a package, where you can freely import and initialize Basilisp.
+
+.. code-block:: python
+
+   import basilisp.main
+
+   basilisp.main.init()
+
+You could also initialize Basilisp in a framework such as Django, where the entrypoint is dictated by the framework.
+For example, you could use a hook such as Django's ``AppConfig.ready()``.
 
 .. code-block:: python
 
diff --git a/src/basilisp/main.py b/src/basilisp/main.py
index ea436d0..542d856 100644
--- a/src/basilisp/main.py
+++ b/src/basilisp/main.py
@@ -1,6 +1,7 @@
 import importlib
 import os
 import sysconfig
+import threading
 from pathlib import Path
 from typing import Optional
 
@@ -11,8 +12,11 @@ from basilisp.lang.compiler import compiler_opts
 from basilisp.lang.typing import CompilerOpts
 from basilisp.lang.util import munge
 
+_INIT_LOCK = threading.Lock()
+_runtime_is_initialized = False
 
-def init(opts: Optional[CompilerOpts] = None) -> None:
+
+def init(opts: Optional[CompilerOpts] = None, force_reload: bool = False) -> None:
     """
     Initialize the runtime environment for Basilisp code evaluation.
 
@@ -22,12 +26,22 @@ def init(opts: Optional[CompilerOpts] = None) -> None:
 
     If you want to execute a Basilisp file which is stored in a well-formed package
     or module structure, you probably want to use :py:func:`bootstrap`.
+
+    ``init()`` may be called more than once. Only the first invocation will initialize
+    the runtime unless ``force_reload=True``.
     """
-    logconfig.configure_root_logger()
-    runtime.init_ns_var()
-    runtime.bootstrap_core(opts if opts is not None else compiler_opts())
-    importer.hook_imports()
-    importlib.import_module("basilisp.core")
+    global _runtime_is_initialized
+
+    with _INIT_LOCK:
+        if _runtime_is_initialized and not force_reload:
+            return
+
+        logconfig.configure_root_logger()
+        runtime.init_ns_var()
+        runtime.bootstrap_core(opts if opts is not None else compiler_opts())
+        importer.hook_imports()
+        importlib.import_module("basilisp.core")
+        _runtime_is_initialized = True
 
 
 def bootstrap(


Test patch:
diff --git a/tests/basilisp/cli_test.py b/tests/basilisp/cli_test.py
index 7072455..53a04bf 100644
--- a/tests/basilisp/cli_test.py
+++ b/tests/basilisp/cli_test.py
@@ -83,6 +83,7 @@ class CapturedIO:
 @pytest.fixture
 def run_cli(monkeypatch, capsys, cap_lisp_io):
     def _run_cli(args: Sequence[str], input: Optional[str] = None):
+        monkeypatch.setattr("basilisp.main._runtime_is_initialized", False)
         if input is not None:
             monkeypatch.setattr(
                 "sys.stdin", io.TextIOWrapper(io.BytesIO(input.encode("utf-8")))