Repository: borgbackup/borg

Issue:
BrokenPipeError leading to a crash when using filtering/counting shell utilities
When using e.g. tail or head to limit the number of lines a BrokenPipeError is thrown and leads to an abnormal exit of borg:

```
% sudo borg list BACKUP | head -n1
20160202-121810                      Tue, 2016-02-02 13:03:51
Local Exception.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1498, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1452, in run
    return args.func(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 80, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 544, in do_list
    print(format_archive(archive_info))
BrokenPipeError: [Errno 32] Broken pipe

Platform: Linux occam 4.6.0-bait.0-wbt #1 SMP PREEMPT Tue May 17 09:32:06 CEST 2016 x86_64 
Linux: debian stretch/sid 
Borg: 1.0.2  Python: CPython 3.5.1+
PID: 21174  CWD: /home/user
sys.argv: ['/usr/bin/borg', 'list', 'BACKUP']
SSH_ORIGINAL_COMMAND: None

Exception ignored in: <_io.TextIOWrapper name='<stdout>' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
```

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/34723987-brokenpipeerror-leading-to-a-crash-when-using-filtering-counting-shell-utilities)



Discussion:
We already had a similar case once and the issue that came up while fixing it was how to differentiate harmless BrokenPipeErrors (like head / tail / more / less) from not harmless ones.

See there: https://github.com/borgbackup/borg/pull/805 (it was not merged - more research / discussion is needed here)

Solving this might need to wrap screen output closely by an exception handler:

```
# bp.py
import sys

for i in range(100000):
    try:
        print(i)
    except BrokenPipeError:
        #might need to close stderr to avoid a warning emitted by python:
        #sys.stderr.close()
        sys.exit(0)
```

Use this like:

```
python3.4 bp.py | head
```

> Solving this might need to wrap screen output closely by an exception handler:

If so, then it should be done after console output was factored out from application logic.

I would like to pick up this bug. I intend to combine suggestions from both @ThomasWaldmann and @enkore.

@andela-kerinoso ok, thanks! just keep us updated via a pull request.

See also the discussion in #1151.

Crazy idea, but would you be fine with patching stdout and stderr to handle a BrokenPipeError by closing the true pipe and ignoring future output?

BTW, I just double checked and that's possible. Example:

``` python
import sys
import io

dev_null = open('/dev/null', 'w')
sys.stdout = io.TextIOWrapper(dev_null)

print("Hello no one!") # won't print anything
```

Oh, looks like we're already wrapping it. I'll see if I can't just switch around some stuff.

@PlasmaPower don't forget to start the bounty / claim it when this is closed.


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -22,7 +22,7 @@
     get_cache_dir, prune_within, prune_split, \
     Manifest, NoManifestError, remove_surrogates, update_excludes, format_archive, check_extension_modules, Statistics, \
     dir_is_tagged, bigint_to_int, ChunkerParams, CompressionSpec, PrefixSpec, is_slow_msgpack, yes, sysinfo, \
-    EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR, log_multi, PatternMatcher
+    EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR, log_multi, PatternMatcher, ErrorIgnoringTextIOWrapper
 from .logger import create_logger, setup_logging
 logger = create_logger()
 from .compress import Compressor, COMPR_BUFFER
@@ -1592,8 +1592,8 @@ def setup_signal_handlers():  # pragma: no cover
 def main():  # pragma: no cover
     # Make sure stdout and stderr have errors='replace') to avoid unicode
     # issues when print()-ing unicode file names
-    sys.stdout = io.TextIOWrapper(sys.stdout.buffer, sys.stdout.encoding, 'replace', line_buffering=True)
-    sys.stderr = io.TextIOWrapper(sys.stderr.buffer, sys.stderr.encoding, 'replace', line_buffering=True)
+    sys.stdout = ErrorIgnoringTextIOWrapper(sys.stdout.buffer, sys.stdout.encoding, 'replace', line_buffering=True)
+    sys.stderr = ErrorIgnoringTextIOWrapper(sys.stderr.buffer, sys.stderr.encoding, 'replace', line_buffering=True)
     setup_signal_handlers()
     archiver = Archiver()
     msg = None
diff --git a/borg/helpers.py b/borg/helpers.py
--- a/borg/helpers.py
+++ b/borg/helpers.py
@@ -12,6 +12,8 @@
 import platform
 import time
 import unicodedata
+import io
+import errno
 
 import logging
 from .logger import create_logger
@@ -1089,3 +1091,27 @@ def log_multi(*msgs, level=logging.INFO):
         lines.extend(msg.splitlines())
     for line in lines:
         logger.log(level, line)
+
+
+class ErrorIgnoringTextIOWrapper(io.TextIOWrapper):
+    def read(self, n):
+        if not self.closed:
+            try:
+                return super().read(n)
+            except BrokenPipeError:
+                try:
+                    super().close()
+                except OSError:
+                    pass
+        return ''
+
+    def write(self, s):
+        if not self.closed:
+            try:
+                return super().write(s)
+            except BrokenPipeError:
+                try:
+                    super().close()
+                except OSError:
+                    pass
+        return len(s)


Test patch: