Repository: datalad/datalad

Issue:
Application vs library use distinction (also Python vs Cmdline)
Seeing https://github.com/datalad/datalad/pull/6078#issuecomment-944040232 reminded me, that we had several cases already where the information whether datalad was called via CLI or python would be desirable.
Currently that is not assessible from within most of the code base. However, the outer layer in `interface`, the arg parsing, etc. do actually have that information. We could simply set a module attribute in `datalad`, `datalad.api` or may be `datalad.interface` to store a flag, that can then be used by code deeper down.

WDYT, @datalad/developers ?


Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/datalad/datalad/pull/6213?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad) Report
> Merging [#6213](https://codecov.io/gh/datalad/datalad/pull/6213?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad) (90c0ffb) into [master](https://codecov.io/gh/datalad/datalad/commit/075fd5d78ef9840e390f4f106fa7ca6c0797475b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad) (075fd5d) will **decrease** coverage by `52.95%`.
> The diff coverage is `50.00%`.

> :exclamation: Current head 90c0ffb differs from pull request most recent head 92324d8. Consider uploading reports for the commit 92324d8 to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/datalad/datalad/pull/6213/graphs/tree.svg?width=650&height=150&src=pr&token=IlkOD0D6oP&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad)](https://codecov.io/gh/datalad/datalad/pull/6213?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad)

```diff
@@             Coverage Diff             @@
##           master    #6213       +/-   ##
===========================================
- Coverage   88.86%   35.90%   -52.96%     
===========================================
  Files         319      320        +1     
  Lines       41648    41735       +87     
===========================================
- Hits        37011    14987    -22024     
- Misses       4637    26748    +22111     
```


| [Impacted Files](https://codecov.io/gh/datalad/datalad/pull/6213?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad) | Coverage Δ | |
|---|---|---|
| [datalad/cmdline/tests/test\_main.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC9jbWRsaW5lL3Rlc3RzL3Rlc3RfbWFpbi5weQ==) | `0.00% <0.00%> (-83.15%)` | :arrow_down: |
| [datalad/downloaders/tests/test\_providers.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC9kb3dubG9hZGVycy90ZXN0cy90ZXN0X3Byb3ZpZGVycy5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [datalad/interface/\_\_init\_\_.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC9pbnRlcmZhY2UvX19pbml0X18ucHk=) | `100.00% <ø> (ø)` | |
| [datalad/interface/run\_procedure.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC9pbnRlcmZhY2UvcnVuX3Byb2NlZHVyZS5weQ==) | `76.33% <ø> (-17.16%)` | :arrow_down: |
| [datalad/interface/tests/test\_results.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC9pbnRlcmZhY2UvdGVzdHMvdGVzdF9yZXN1bHRzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [datalad/interface/unlock.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC9pbnRlcmZhY2UvdW5sb2NrLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [datalad/local/tests/test\_unlock.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC9sb2NhbC90ZXN0cy90ZXN0X3VubG9jay5weQ==) | `0.00% <0.00%> (ø)` | |
| [datalad/support/external\_versions.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC9zdXBwb3J0L2V4dGVybmFsX3ZlcnNpb25zLnB5) | `74.35% <ø> (-20.52%)` | :arrow_down: |
| [datalad/tests/test\_cmd.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC90ZXN0cy90ZXN0X2NtZC5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [datalad/tests/test\_utils.py](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad#diff-ZGF0YWxhZC90ZXN0cy90ZXN0X3V0aWxzLnB5) | `0.00% <0.00%> (-97.18%)` | :arrow_down: |
| ... and [271 more](https://codecov.io/gh/datalad/datalad/pull/6213/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/datalad/datalad/pull/6213?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/datalad/datalad/pull/6213?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad). Last update [075fd5d...92324d8](https://codecov.io/gh/datalad/datalad/pull/6213?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=datalad).

bpoldrack: I like it. As far as I'm concerned: Let's get some experience with this before (if at all) developing something more complex.
mih: Conclusion from call: Replace the entire API with config manager calls. Do inheritance to subprocesses via ENV variables.
mih: Despite the conclusion from the call yesterday, I did not base the facility on the configmanager entirely. It caused too many ripples, import dependencies and expensive checks. That being said, the cmdline switch `--library-mode` is gone (no longer needed).

Patch:
diff --git a/datalad/__init__.py b/datalad/__init__.py
index 03a636039..06be087fa 100644
--- a/datalad/__init__.py
+++ b/datalad/__init__.py
@@ -23,6 +23,66 @@ import atexit
 import os
 
 
+# this is not to be modified. for querying use get_apimode()
+__api = 'python'
+
+
+def get_apimode():
+    """Returns the API mode label for the current session.
+
+    The API mode label indicates whether DataLad is running in "normal"
+    mode in a Python session, or whether it is used via the command line
+    interface.
+
+    This function is a utility for optimizing behavior and messaging to the
+    particular API (Python vs command line) in use in a given process.
+
+    Returns
+    {'python', 'cmdline'}
+      The API mode is 'python' by default, unless the main command line
+      entrypoint set it to 'cmdline'.
+    """
+    return __api
+
+
+def in_librarymode():
+    """Returns whether DataLad is requested to run in "library mode"
+
+    In this mode DataLad aims to behave without the assumption that it is
+    itself the front-end of a process and in full control over messaging
+    and parameters.
+
+    Returns
+    -------
+    bool
+    """
+    return __runtime_mode == 'library'
+
+
+def enable_librarymode():
+    """Request DataLad to operate in library mode.
+
+    This function should be executed immediately after importing the `datalad`
+    package, when DataLad is not used as an application, or in interactive
+    scenarios, but as a utility library inside other applications. Enabling
+    this mode will turn off some convenience feature that are irrelevant in
+    such use cases (with performance benefits), and alters it messaging
+    behavior to better interoperate with 3rd-party front-ends.
+
+    Library mode can only be enabled once. Switching it on and off within
+    the runtime of a process is not supported.
+
+    Example::
+
+        >>> import datalad
+        >>> datalad.enable_librarymode()
+    """
+    global __runtime_mode
+    __runtime_mode = 'library'
+    # export into the environment for child processes to inherit
+    os.environ['DATALAD_RUNTIME_LIBRARYMODE'] = '1'
+
+
 # For reproducible demos/tests
 _seed = os.environ.get('DATALAD_SEED', None)
 if _seed is not None:
@@ -50,6 +110,13 @@ from .config import ConfigManager
 cfg = ConfigManager()
 
 # must come after config manager
+# this is not to be modified. see enable/in_librarymode()
+# by default, we are in application-mode, simply because most of
+# datalad was originally implemented with this scenario assumption
+__runtime_mode = 'library' \
+    if cfg.getbool('datalad.runtime', 'librarymode', False) \
+    else 'application'
+
 from .log import lgr
 from datalad.support.exceptions import CapturedException
 from datalad.utils import (
diff --git a/datalad/api.py b/datalad/api.py
index 80b8b1db4..ced4e7680 100644
--- a/datalad/api.py
+++ b/datalad/api.py
@@ -8,6 +8,7 @@
 # ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
 """Python DataLad API exposing user-oriented commands (also available via CLI)"""
 
+import datalad
 from datalad.coreapi import *
 
 
@@ -35,7 +36,8 @@ def _command_summary():
     return "\n".join(get_cmd_summaries(grp_short_descriptions, groups))
 
 
-__doc__ += "\n\n{}".format(_command_summary())
+if not datalad.in_librarymode():
+    __doc__ += "\n\n{}".format(_command_summary())
 
 
 def _generate_extension_api():
diff --git a/datalad/cmdline/main.py b/datalad/cmdline/main.py
index fc01a1803..4e5abcd06 100644
--- a/datalad/cmdline/main.py
+++ b/datalad/cmdline/main.py
@@ -151,6 +151,8 @@ def setup_parser(
 
 def main(args=None):
     lgr.log(5, "Starting main(%r)", args)
+    # record that we came in via the cmdline
+    datalad.__api = 'cmdline'
     args = args or sys.argv
     if on_msys_tainted_paths:
         # Possibly present DataLadRIs were stripped of a leading /
@@ -188,6 +190,8 @@ def main(args=None):
 
     # enable overrides
     datalad.cfg.reload(force=True)
+    if 'datalad.runtime.librarymode' in datalad.cfg:
+        datalad.enable_librarymode()
 
     if cmdlineargs.change_path is not None:
         from .common_args import change_path as change_path_opt
@@ -261,6 +265,8 @@ def main(args=None):
         ce = CapturedException(exc)
         lgr.error("Failed to render results due to %s", ce)
         sys.exit(1)
+    # all good, not strictly needed, but makes internal testing easier
+    sys.exit(0)
 
 
 lgr.log(5, "Done importing cmdline.main")
diff --git a/datalad/interface/base.py b/datalad/interface/base.py
index 10f4a7669..40186a26e 100644
--- a/datalad/interface/base.py
+++ b/datalad/interface/base.py
@@ -30,6 +30,7 @@ import warnings
 
 from ..ui import ui
 
+import datalad
 from datalad.interface.common_opts import eval_params
 from datalad.interface.common_opts import eval_defaults
 from datalad.support.constraints import (
@@ -484,6 +485,9 @@ def build_doc(cls, **kwargs):
     cls: Interface
       class defining a datalad command
     """
+    if datalad.in_librarymode():
+        lgr.debug("Not assembling DataLad API docs in libary-mode")
+        return cls
 
     # Note, that this is a class decorator, which is executed only once when the
     # class is imported. It builds the docstring for the class' __call__ method
diff --git a/docs/source/design/application_vs_library_mode.rst b/docs/source/design/application_vs_library_mode.rst
new file mode 100644
index 000000000..3b9d3d12a
--- /dev/null
+++ b/docs/source/design/application_vs_library_mode.rst
@@ -0,0 +1,59 @@
+.. -*- mode: rst -*-
+.. vi: set ft=rst sts=4 ts=4 sw=4 et tw=79:
+
+.. _chap_design_application_vs_libary_mode:
+
+***************************************
+Application-type vs. library-type usage
+***************************************
+
+.. topic:: Specification scope and status
+
+   This specification describes the current implementation.
+
+Historically, DataLad was implemented with the assumption of application-type
+usage, i.e., a person using DataLad through any of its APIs. Consequently,
+(error) messaging was primarily targeting humans, and usage advice focused on
+interactive use. With the increasing utilization of DataLad as an
+infrastructural component it was necessary to address use cases of library-type
+or internal usage more explicitly.
+
+DataLad continues to behave like a stand-alone application by default.
+
+For internal use, Python and command-line APIs provide dedicated mode switches.
+
+Library mode can be enabled by setting the boolean configuration setting
+``datalad.runtime.librarymode`` **before the start of the DataLad process**.
+From the command line, this can be done with the option
+``-c datalad.runtime.librarymode=yes``, or any other means for setting
+configuration. In an already running Python process, library mode can be
+enabled by calling ``datalad.enable_libarymode()``. This should be done
+immediately after importing the ``datalad`` package for maximum impact.
+
+.. code-block:: python
+
+   >>> import datalad
+   >>> datalad.enable_libarymode()
+
+In a Python session, library mode **cannot** be enabled reliably by just setting
+the configuration flag **after** the ``datalad`` package was already imported.
+The ``enable_librarymode()`` function must be used.
+
+Moreover, with ``datalad.in_librarymode()`` a query utility is provided that
+can be used throughout the code base for adjusting behavior according to the
+usage scenario.
+
+Switching back and forth between modes during the runtime of a process is not
+supported.
+
+A library mode setting is exported into the environment of the Python process.
+By default, it will be inherited by all child-processes, such as dataset
+procedure executions.
+
+
+Library-mode implications
+=========================
+
+No Python API docs
+  Generation of comprehensive doc-strings for all API commands is skipped. This
+  speeds up ``import datalad.api`` by about 30%.
diff --git a/docs/source/design/index.rst b/docs/source/design/index.rst
index 5c8721666..d8fd2eae1 100644
--- a/docs/source/design/index.rst
+++ b/docs/source/design/index.rst
@@ -13,6 +13,7 @@ subsystems in DataLad.
 .. toctree::
    :maxdepth: 2
 
+   application_vs_library_mode
    file_url_handling
    result_records
    dataset_argument


Test patch:
diff --git a/datalad/cmdline/tests/test_main.py b/datalad/cmdline/tests/test_main.py
index 78fbdf848..d7806cd8b 100644
--- a/datalad/cmdline/tests/test_main.py
+++ b/datalad/cmdline/tests/test_main.py
@@ -45,6 +45,7 @@ from datalad.tests.utils import (
     assert_re_in,
     eq_,
     in_,
+    ok_,
     ok_startswith,
     on_windows,
     slow,
@@ -70,21 +71,28 @@ def run_main(args, exit_code=0, expect_stderr=False):
     stdout, stderr  strings
        Output produced
     """
-    with patch('sys.stderr', new_callable=StringIO) as cmerr:
-        with patch('sys.stdout', new_callable=StringIO) as cmout:
-            with assert_raises(SystemExit) as cm:
-                main(["datalad"] + list(args))
-            assert_equal(cm.exception.code, exit_code)
-            stdout = cmout.getvalue()
-            stderr = cmerr.getvalue()
-            if expect_stderr is False:
-                assert_equal(stderr, "")
-            elif expect_stderr is True:
-                # do nothing -- just return
-                pass
-            else:
-                # must be a string
-                assert_equal(stderr, expect_stderr)
+    was_mode = datalad.__api
+    try:
+        with patch('sys.stderr', new_callable=StringIO) as cmerr:
+            with patch('sys.stdout', new_callable=StringIO) as cmout:
+                with assert_raises(SystemExit) as cm:
+                    main(["datalad"] + list(args))
+                eq_('cmdline', datalad.get_apimode())
+                assert_equal(cm.exception.code, exit_code)
+                stdout = cmout.getvalue()
+                stderr = cmerr.getvalue()
+                if expect_stderr is False:
+                    assert_equal(stderr, "")
+                elif expect_stderr is True:
+                    # do nothing -- just return
+                    pass
+                else:
+                    # must be a string
+                    assert_equal(stderr, expect_stderr)
+    finally:
+        # restore what we had
+        datalad.__api = was_mode
+
     return stdout, stderr
 
 
@@ -379,3 +387,18 @@ def test_commanderror_jsonmsgs(src, exp):
             protocol=StdOutErrCapture)
     if ds.repo.git_annex_version >= "8.20201129":
         in_('use `git-annex export`', cme.exception.stderr)
+
+
+def test_librarymode():
+    was_mode = datalad.__runtime_mode
+    try:
+        # clean --dry-run is just a no-op command that is cheap
+        # to execute. It has no particular role here, other than
+        # to make the code pass the location where library mode
+        # should be turned on via the cmdline API
+        run_main(['-c', 'datalad.runtime.librarymode=yes', 'clean', '--dry-run'])
+        ok_(datalad.in_librarymode())
+    finally:
+        # restore pre-test behavior
+        datalad.__runtime_mode = was_mode
+        datalad.cfg.overrides.pop('datalad.runtime.librarymode')
diff --git a/datalad/support/tests/test_sshrun.py b/datalad/support/tests/test_sshrun.py
index fce4826b5..19d905323 100644
--- a/datalad/support/tests/test_sshrun.py
+++ b/datalad/support/tests/test_sshrun.py
@@ -77,8 +77,9 @@ def test_ssh_option():
     # back an empty value, assume that isn't configured, and skip the test.
     with patch.dict('os.environ', {"LC_DATALAD_HACK": 'hackbert'}):
         with swallow_outputs() as cmo:
-            main(["datalad", "sshrun", "-oSendEnv=LC_DATALAD_HACK",
-                  "datalad-test", "echo $LC_DATALAD_HACK"])
+            with assert_raises(SystemExit):
+                main(["datalad", "sshrun", "-oSendEnv=LC_DATALAD_HACK",
+                      "datalad-test", "echo $LC_DATALAD_HACK"])
             out = cmo.out.strip()
             if not out:
                 raise SkipTest(