Repository: borgbackup/borg

Issue:
Define a public API
(Note: This issue is phrased assertively to fit into the format on a Issue Tracker. This is by no means an attempt to force anything onto the project.)

Borg should have a public, stable and documented API for other tools to consume. This can be either by defining public python APIs or a pipe-based interface with borg commands.

This would faciliate the creation of wrapper tools which work with borg, such as GUIs and anything beyond simple cronjobs.



Discussion:
The api shouldn't be considered stable yet. We'll do that when the time has come.

Has the time come now since v1.0 has been released? :smile:

Completely agree.
pip is at version 9 and has no public api

please keep in mind that even a minor mistake in public api can kill immense potential for future development
Another facet is that you can absolutely write stuff against Borg right now already, *but* that means that *you* are responsible for testing and keeping it working, and also for understanding it since there are no official docs or guides. Declaring a stable API would mainly mean that this work is shifted from others to the Borg project (ie. *we* are responsible for not breaking stuff and checking that it still works exactly the same way), plus having to maintain docs for the API and guides and all.

I'm just gonna say it - I'd rather work on other stuff, especially when it comes to docs. Most use cases are well catered-for by the command line, which we *is* stable in the same release series.

Sometimes it would be handy to have machine readable output, which is something that is being worked on and other tickets exist for that.
On the other hand, the fact that there is no Public API at all makes it hard to develop more user-friendly front-ends for Borg.

Currently I’m invoking borg on the commandline and parsing the output, but that is … not very great, to be honest. With some trickery you can get some commands to provide machine-readable output, but its really not great.

Okay, if you are worried about having to maintain it, maybe there could be a middle-ground. Provide developers with machine-friendly entry points, either in the form of providing a module name (and a link to the source code from which the developer can learn how to use it) or JSON/python dict output on the CLI. Make it *very* clear that the API may change without prior notice and that consumers of the API are responsible for making sure it works as intended. That would already be a huge improvement over having to parse the command line output, which is a mess to do.
@RonnyPfannschmidt not really. Minor mistakes get corrected and endpoints deprecated, it's really not a big deal at all. Especially if you release a public API and state that it is unstable until, say, v2.0, and use it at your peril.

@horazont this. I've been wanting to write a simple and similar UI for macOS like Time Machine but there's currently no reasonable way to do this. I could then also hook up any alerts or errors to the notifications API and have it feel like a native part of the OS.
> Especially if you release a public API and state that it is unstable until, say, v2.0, and use it at your peril.

Which would be the same situation as right now, just that someone sat down and wrote docs for it (and maintains them), right?
@enkore maybe, as I haven't seen the docs. ;)

It is probable that if there were docs that a wrapper that makes building a UI around it easier would be feasible, yes.
@enkore Okay, I see your point—you don’t want to spend the time on docs for an API which is unstable. Fair. Also I have dug into the borg source code, and it is very readable and easy to find the entry points one would need. So, as you said, we already have the state which I suggested (I would have actually preferred a CLI-based API, but directly performing calls to the python call is also fine with me).
there could be a merit in declaring a outer facing api that resembles the cli but exposes specially defined datastructures
Things like better machine readable output for the CLI are coming.

Internally there are the do_XYZ things in archiver.py which take their `args` which are declared in the `build_parser` function far below (actually that's half the file due to the many help texts :D) ~~ there also has been ideas to improve this a bit eg. #795, making things more fine-grained. Eg. one class per command and more than one method does work. This could be a more usable API for programmatic invocation.

---

If we're talking about a public API I get the feeling that not everyone means the same thing.  I mostly mean a stable, documented *Python API* (classes, functions). A *CLI-based API* would mean the current command line interface + machine readable output?
a python api that frees from the need of parsing may be helpful, for as long as it avoids leaking the internals

as soon as the internals leak any change can break downstreams

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -97,6 +97,7 @@ def show_progress(self, item=None, final=False, stream=None, dt=None):
             if self.output_json:
                 data = self.as_dict()
                 data.update({
+                    'time': time.time(),
                     'type': 'archive_progress',
                     'path': remove_surrogates(item.path if item else ''),
                 })
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -1283,7 +1283,8 @@ def is_slow_msgpack():
 def yes(msg=None, false_msg=None, true_msg=None, default_msg=None,
         retry_msg=None, invalid_msg=None, env_msg='{} (from {})',
         falsish=FALSISH, truish=TRUISH, defaultish=DEFAULTISH,
-        default=False, retry=True, env_var_override=None, ofile=None, input=input, prompt=True):
+        default=False, retry=True, env_var_override=None, ofile=None, input=input, prompt=True,
+        msgid=None):
     """Output <msg> (usually a question) and let user input an answer.
     Qualifies the answer according to falsish, truish and defaultish as True, False or <default>.
     If it didn't qualify and retry is False (no retries wanted), return the default [which
@@ -1313,6 +1314,22 @@ def yes(msg=None, false_msg=None, true_msg=None, default_msg=None,
     :param input: input function [input from builtins]
     :return: boolean answer value, True or False
     """
+    def output(msg, msg_type, is_prompt=False, **kwargs):
+        json_output = getattr(logging.getLogger('borg'), 'json', False)
+        if json_output:
+            kwargs.update(dict(
+                type='question_%s' % msg_type,
+                msgid=msgid,
+                message=msg,
+            ))
+            print(json.dumps(kwargs), file=sys.stderr)
+        else:
+            if is_prompt:
+                print(msg, file=ofile, end='', flush=True)
+            else:
+                print(msg, file=ofile)
+
+    msgid = msgid or env_var_override
     # note: we do not assign sys.stderr as default above, so it is
     # really evaluated NOW,  not at function definition time.
     if ofile is None:
@@ -1320,13 +1337,13 @@ def yes(msg=None, false_msg=None, true_msg=None, default_msg=None,
     if default not in (True, False):
         raise ValueError("invalid default value, must be True or False")
     if msg:
-        print(msg, file=ofile, end='', flush=True)
+        output(msg, 'prompt', is_prompt=True)
     while True:
         answer = None
         if env_var_override:
             answer = os.environ.get(env_var_override)
             if answer is not None and env_msg:
-                print(env_msg.format(answer, env_var_override), file=ofile)
+                output(env_msg.format(answer, env_var_override), 'env_answer', env_var=env_var_override)
         if answer is None:
             if not prompt:
                 return default
@@ -1337,23 +1354,23 @@ def yes(msg=None, false_msg=None, true_msg=None, default_msg=None,
                 answer = truish[0] if default else falsish[0]
         if answer in defaultish:
             if default_msg:
-                print(default_msg, file=ofile)
+                output(default_msg, 'accepted_default')
             return default
         if answer in truish:
             if true_msg:
-                print(true_msg, file=ofile)
+                output(true_msg, 'accepted_true')
             return True
         if answer in falsish:
             if false_msg:
-                print(false_msg, file=ofile)
+                output(false_msg, 'accepted_false')
             return False
         # if we get here, the answer was invalid
         if invalid_msg:
-            print(invalid_msg, file=ofile)
+            output(invalid_msg, 'invalid_answer')
         if not retry:
             return default
         if retry_msg:
-            print(retry_msg, file=ofile, end='', flush=True)
+            output(retry_msg, 'prompt_retry', is_prompt=True)
         # in case we used an environment variable and it gave an invalid answer, do not use it again:
         env_var_override = None
 
@@ -1432,7 +1449,8 @@ def output_json(self, *, finished=False, **kwargs):
             operation=self.id,
             msgid=self.msgid,
             type=self.JSON_TYPE,
-            finished=finished
+            finished=finished,
+            time=time.time(),
         ))
         print(json.dumps(kwargs), file=sys.stderr)
 
diff --git a/src/borg/logger.py b/src/borg/logger.py
--- a/src/borg/logger.py
+++ b/src/borg/logger.py
@@ -204,7 +204,6 @@ def critical(self, *args, **kw):
 
 class JsonFormatter(logging.Formatter):
     RECORD_ATTRIBUTES = (
-        'created',
         'levelname',
         'name',
         'message',
@@ -224,6 +223,7 @@ def format(self, record):
         super().format(record)
         data = {
             'type': 'log_message',
+            'time': record.created,
         }
         for attr in self.RECORD_ATTRIBUTES:
             value = getattr(record, attr, None)


Test patch: