Repository: borgbackup/borg

Issue:
use ctrl-c to trigger checkpoint&abort
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Issue, really a feature request.

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

1.1.9

#### Operating system (distribution) and version.

Ubuntu 16.04, Ubuntu 18.04, CentOS 7, OS X 10.13.6.

#### Hardware / network configuration, and filesystems used.

Various.

#### How much data is handled by borg?

Varies, kBytes to GBytes.

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

    borg create --progress ...

## Describe the problem you're observing.

Borg makes checkpoints at intervals. Suppose we want to interrupt a backup (maybe it's taking too long and we'll try again later). If waiting 5 minutes longer would allow a checkpoint to be created, it's advisable to wait. But Borg doesn't tell us when a checkpoint is coming or when one has been done. So we have no idea when is the right time to hit ^C.

A good solution would be for Borg to print when a checkpoint has been done if the --progress option has been given; and, if possible, also print how much longer to the next checkpoint.

In a more fanciful scenario, I can imagine a progress bar with marks showing where checkpoints will occur or have occurred. Like how some video services show you where the ads are. (May be hard to implement as Borg probably does not know how much data will be transmitted in the future.)

Currently, --progress causes a line of output to be shown at the bottom of the screen. The documentation doesn't mention that any part of this output tells us when a checkpoint occurs. 

Admittedly, I have not downloaded and tested version 1.1.10 to see if it prints anything different.

A search for the word "checkpoint" in issues, docs, and faq didn't turn up anything relevant.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Not applicable.

#### Include any warning/errors/backtraces from the system logs

Not applicable.


<!--

If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html

*IMPORTANT* - Please mark logs and text output from terminal commands 
or else Github will not display them correctly. 
An example is provided below.

Example:
```
this is an example how log text should be marked (wrap it with ```)
```
-->



Discussion:
Is it really that important?

The default is 1800s (30mins) IIRC, so in the medium case you'ld lose about 15mins, in the worst case almost 30mins.

The default used to be less (5mins IIRC) in earlier borg/attic versions, but that had a big impact due to the overhead of a commit, so it was re-adjusted to default to 30mins to have less overhead / better overall speed.

If you expect to interrupt often or to have a rather unstable repo connection, you can reduce the checkpoint interval to less than 30mins by using the commandline option.

About the progress display:
- No, it doesn't show when a checkpoint happened / will happen.
- I'm a bit reluctant about adding / changing the progress display. it is rather crowded already due to amount of information it shows. it needs to fit into 80 char width and adding anything reduces the space available for the path/filename.
- Users often wonder what the info in the progress display means and adding even more infos wouldn't make it easier.

I think most people would find it useful. I know I would have when I was debugging backup scripts sending data on a limited-speed upllink and using ^C frequently.

It might be only 15 min of data on the average, but human psychology makes it seem a lot more, when you have an annoying suspicion that you just lost 29 minutes of backup effort. Not knowing makes it seem a lot worse.

The real question perhaps is how much we are willing to give up to get checkpoint status. The current --progress display stays on one line, so it doesn't pollute the screen by filling it with lines of output. On the down side, we only get to see current information, and it's hard to tell how backup speed varies with time.

If there were to be a more verbose display that actually filled the screen with lines of output, then including checkpoint status in that would be a good idea. The possibility of producing such additional verbose output (perhaps with a --verbose option) should be considered. Maybe for the long run.
The other possibility would be a two-line display instead of one line. I truly do not know what it takes in coding effort to do this. Can one trivially generalize to n lines of status, or does increasing the lines beyond 1 take entirely different code?
2 line display is not possible using the super-simplistic way of cursor control we use now (just using CR).
Here are a couple of ideas.

Have a certain signal, e.g., SIGQUIT, set a flag.

Elsewhere, in a suitable loop, when that flag is seen, a checkpoint is done and then the program exits.

Alternatively:

Most of the time when Borg is manually interrupted, the user will resume eventually. So maybe instead of SIGQUIT we just use SIGINT. Then a single ^C should always cause a checkpoint-and-exit as above. A second ^C during the checkpoint can then tell Borg to abort the checkpoint and do an immediate exit.

@fortran77 sounds neat, but guess this could produce quite complicated tracebacks and make analysis more complicated when Ctrl-C is hit in a error/hanging state (because trying to commit then would just error/hang again).
Direct response is at the end below. My thought process comes first:

My first inclination was to respond by recommending my first idea. I.e., SIGINT causes Borg to exit, while SIGQUIT asks for checkpoint-then-exit. This way nothing changes for the average user who hits ^C when trying to abort a hung backup.

But then I thought: Why make the user have to think about whether or not a checkpoint should be done? We should just try to do the checkpoint on ^C. That way, if a checkpoint is at all possible, one will be done.

The average user will notice if ^C isn't interrupting the backup, and will surely hit ^C a few more times We are all used to misbehaving processes not exiting, and it's normal for us to try again a few times.

So my second idea seems the better one. If Borg is behaving properly but the network is hung, it will try a checkpoint and remain hung, and the user will hit ^C again and Borg will exit.

If Borg is behaving properly but the backup is just going slowly, the user will hit ^C, a checkpoint will occur, and everything will be fine. If a message could be printed "Saving checkpoint ...", the user will know to wait a little for this to complete before hitting ^C again.

Now to respond more directly to your comment: The first ^C sets a flag **and disables the ^C signal handler**. The flag is detected in a loop elsewhere, causing a checkpoint-then-exit. 

So the second ^C acts normally and interrupts whatever is going on. None of the code, except setting the flag, will occur inside a signal handler. So **the only effect of the first ^C is to set a flag.** All other code executes normally.

If the code is misbehaving, it might not detect the flag at all. The second ^C will cause it to exit. If it does detect the flag and begins a commit, then the second ^C will abort the commit.

So from the user's point of view, the second ^C acts just like the first ^C used to act in the past.
OK, sounds good, guess we'll have to try that.
Great! I am glad my somewhat long explanation made sense.

I'll start learning Python, just in case.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -40,6 +40,7 @@
 from .helpers import ellipsis_truncate, ProgressIndicatorPercent, log_multi
 from .helpers import os_open, flags_normal
 from .helpers import msgpack
+from .helpers import sig_int
 from .patterns import PathPrefixPattern, FnmatchPattern, IECommand
 from .item import Item, ArchiveItem, ItemDiff
 from .platform import acl_get, acl_set, set_flags, get_flags, swidth, hostname
@@ -1095,6 +1096,19 @@ def write_part_file(self, item, from_chunk, number):
         self.write_checkpoint()
         return length, number
 
+    def maybe_checkpoint(self, item, from_chunk, part_number, forced=False):
+        sig_int_triggered = sig_int and sig_int.action_triggered()
+        if forced or sig_int_triggered or \
+            self.checkpoint_interval and time.monotonic() - self.last_checkpoint > self.checkpoint_interval:
+            if sig_int_triggered:
+                logger.info('checkpoint requested: starting checkpoint creation...')
+            from_chunk, part_number = self.write_part_file(item, from_chunk, part_number)
+            self.last_checkpoint = time.monotonic()
+            if sig_int_triggered:
+                sig_int.action_completed()
+                logger.info('checkpoint requested: finished checkpoint creation!')
+        return from_chunk, part_number
+
     def process_file_chunks(self, item, cache, stats, show_progress, chunk_iter, chunk_processor=None):
         if not chunk_processor:
             def chunk_processor(data):
@@ -1113,17 +1127,14 @@ def chunk_processor(data):
             item.chunks.append(chunk_processor(data))
             if show_progress:
                 stats.show_progress(item=item, dt=0.2)
-            if self.checkpoint_interval and time.monotonic() - self.last_checkpoint > self.checkpoint_interval:
-                from_chunk, part_number = self.write_part_file(item, from_chunk, part_number)
-                self.last_checkpoint = time.monotonic()
+            from_chunk, part_number = self.maybe_checkpoint(item, from_chunk, part_number, forced=False)
         else:
             if part_number > 1:
                 if item.chunks[from_chunk:]:
                     # if we already have created a part item inside this file, we want to put the final
                     # chunks (if any) into a part item also (so all parts can be concatenated to get
                     # the complete file):
-                    from_chunk, part_number = self.write_part_file(item, from_chunk, part_number)
-                    self.last_checkpoint = time.monotonic()
+                    from_chunk, part_number = self.maybe_checkpoint(item, from_chunk, part_number, forced=True)
 
                 # if we created part files, we have referenced all chunks from the part files,
                 # but we also will reference the same chunks also from the final, complete file:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -71,6 +71,7 @@
     from .helpers import umount
     from .helpers import flags_root, flags_dir, flags_special_follow, flags_special
     from .helpers import msgpack
+    from .helpers import sig_int
     from .nanorst import rst_to_terminal
     from .patterns import ArgparsePatternAction, ArgparseExcludeFileAction, ArgparsePatternFileAction, parse_exclude_pattern
     from .patterns import PatternMatcher
@@ -531,7 +532,12 @@ def create_inner(archive, cache, fso):
                 if args.progress:
                     archive.stats.show_progress(final=True)
                 archive.stats += fso.stats
-                archive.save(comment=args.comment, timestamp=args.timestamp, stats=archive.stats)
+                if sig_int:
+                    # do not save the archive if the user ctrl-c-ed - it is valid, but incomplete.
+                    # we already have a checkpoint archive in this case.
+                    self.print_error("Got Ctrl-C / SIGINT.")
+                else:
+                    archive.save(comment=args.comment, timestamp=args.timestamp, stats=archive.stats)
                 args.stats |= args.json
                 if args.stats:
                     if args.json:
@@ -587,6 +593,10 @@ def _process(self, *, path, parent_fd=None, name=None,
 
         This should only raise on critical errors. Per-item errors must be handled within this method.
         """
+        if sig_int and sig_int.action_done():
+            # the user says "get out of here!" and we have already completed the desired action.
+            return
+
         try:
             recurse_excluded_dir = False
             if matcher.match(path):
@@ -4431,7 +4441,8 @@ def main():  # pragma: no cover
                 print(tb, file=sys.stderr)
             sys.exit(e.exit_code)
         try:
-            exit_code = archiver.run(args)
+            with sig_int:
+                exit_code = archiver.run(args)
         except Error as e:
             msg = e.get_message()
             msgid = type(e).__qualname__
diff --git a/src/borg/helpers/process.py b/src/borg/helpers/process.py
--- a/src/borg/helpers/process.py
+++ b/src/borg/helpers/process.py
@@ -86,6 +86,52 @@ def handler(sig_no, frame):
     return handler
 
 
+class SigIntManager:
+    def __init__(self):
+        self._sig_int_triggered = False
+        self._action_triggered = False
+        self._action_done = False
+        self.ctx = signal_handler('SIGINT', self.handler)
+
+    def __bool__(self):
+        # this will be True (and stay True) after the first Ctrl-C/SIGINT
+        return self._sig_int_triggered
+
+    def action_triggered(self):
+        # this is True to indicate that the action shall be done
+        return self._action_triggered
+
+    def action_done(self):
+        # this will be True after the action has completed
+        return self._action_done
+
+    def action_completed(self):
+        # this must be called when the action triggered is completed,
+        # to avoid that the action is repeatedly triggered.
+        self._action_triggered = False
+        self._action_done = True
+
+    def handler(self, sig_no, stack):
+        # handle the first ctrl-c / SIGINT.
+        self.__exit__(None, None, None)
+        self._sig_int_triggered = True
+        self._action_triggered = True
+
+    def __enter__(self):
+        self.ctx.__enter__()
+
+    def __exit__(self, exception_type, exception_value, traceback):
+        # restore the original ctrl-c handler, so the next ctrl-c / SIGINT does the normal thing:
+        if self.ctx:
+            self.ctx.__exit__(exception_type, exception_value, traceback)
+            self.ctx = None
+
+
+# global flag which might trigger some special behaviour on first ctrl-c / SIGINT,
+# e.g. if this is interrupting "borg create", it shall try to create a checkpoint.
+sig_int = SigIntManager()
+
+
 def popen_with_error_handling(cmd_line: str, log_prefix='', **kwargs):
     """
     Handle typical errors raised by subprocess.Popen. Return None if an error occurred,


Test patch: