Repository: borgbackup/borg

Issue:
RemoteArchiverTestCase failures on cygwin (1.0-maint)
Quite some of these tests either hang or fail:

```
py.test -v -k Remote
```

The corresponding local tests work:

```
py.test -v -k "not Remote"    # ... and not bench
```

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/35765312-remotearchivertestcase-failures-on-cygwin-1-0-maint)



Discussion:
It seems like borg.remote receives crap via the inter-process pipes.
The msgpack unpacker does not unpack the expected data formats.

It does not seem to be related to "text vs binary mode", I did some experiments in a cygwin python interpreter and was able to send b'\n\001\002\000' to a subprocess and it was echoed back unchanged.

So, what can it be? Can a pipe lose data?

Hello. I was unlucky enough to have my backup server running windows . So i was hit by this bug using cygwin package.I could create archive without problem, but cannot get info or mount it on my linux server. I narrowed it down to  pipe buffer size on cygwin.I was able to resolve it by patching cygwin dll and increasing pipe default buffer.

https://gist.github.com/skabanov/bf23e097f3e3057e4a3f89f897e3e3d5

Added by TW:

https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=commitdiff;h=41045fb74d9413be655838aef5a6089bc6a229d5

This sounds a lot like a (bad) bug in cygwin. Did you / can you raise this issue with the cygwin project?

Out of interest: What errors did you get on Linux? Does reading (info, list, check) work on Windows?

Locally on windows everything is working.Remotely list, checks, create are ok.For info and mount
i got either:
Local Exception.
Traceback (most recent call last):
  File "borg/archiver.py", line 83, in wrapper
  File "borg/archiver.py", line 95, in wrapper
  File "borg/archiver.py", line 591, in do_info
  File "borg/archive.py", line 345, in calc_stats
  File "borg/remote.py", line 404, in get_many
  File "borg/remote.py", line 329, in call_many
Exception: Unexpected RPC data format.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "borg/archiver.py", line 1684, in main
  File "borg/archiver.py", line 1621, in run
  File "borg/archiver.py", line 85, in wrapper
  File "borg/remote.py", line 226, in **exit**
  File "borg/remote.py", line 388, in rollback
  File "borg/remote.py", line 271, in call
  File "borg/remote.py", line 329, in call_many
Exception: Unexpected RPC data format.

or just hunging pipe.

Looks like they reverted big pipe sizes in 2011 because of this:
http://cygwin.com/ml/cygwin/2011-03/msg00541.html

Ah right, my bad. This is about RPC issues, I thought this was about writing to disk.

@skabanov can you discuss this with the cygwin developers?

https://cygwin.com/problems.html

I recently ran the tests in the new windows10 vagrant machine and the RemoteArchiver / RemoteRepository issues seems the be the last big problem, the other tests work fine.

http://hastebin.com/wirabaropo.sql seen by ChainFire in practice - same cause?
I've tagged it with corruption, as it seems that borg data gets corrupted between borg client and borg repo server. Likely not our fault, but still interesting for potential users of borg on cygwin.
Considering above issue and if running under windows 10, #1961 might be interesting.
Note: I send a notice to the `cygwin` mailing list: https://cygwin.com/ml/cygwin/2017-01/msg00087.html

https://github.com/borgbackup/borg/blob/1.0.9/borg/remote.py#L188 this is the place where the ssh subprocess is forked and the pipes are set up.

Python docs say:
```
If universal_newlines is False the file objects stdin, stdout and stderr will be
opened as binary streams, and no line ending conversion is done.
```
False is the default value of that param.
Just noticed that above is only the borg client pipe setup. The borg serve setup is here:

https://github.com/borgbackup/borg/blob/1.0.9/borg/remote.py#L73

And here is maybe the culprit: the stdin/stdout/stderr likely defaults to text mode and really needs to be explicitly set to binary on windows for serve().
If this theory is correct it would mean that the FDs are in text mode, so \r\n things get mangled, which seems a plausible cause. _setmode() would fix that.

Python [has that](https://docs.python.org/3/library/msvcrt.html#msvcrt.setmode) but I'm not sure whether that module is available in cygwin, since it's called msvcrt. Otherwise we would need to create a cygwin platform module and add a platform API to do this (`force_binary_fd` or so).

Cygwin ref: https://cygwin.com/cygwin-ug-net/using-textbinary.html
Hmm, we use 2 kinds of I/O there:
- logging (to stderr, should not matter for this corruption issue)
- os.read(), os.write() to the stdin/stdout FDs (no text mode issue with that!)
@skabanov - the issue about pipes you cited was on WinXP where total user member was limited to 2G (or 3G in some special cases).  WinXP is no longer supported and certainly most users will have more than a 2G address space.

@ThomasWaldmann - I don't think text mode is your problem -- it doesn't make sense to use text mode for a pipe, not to mention, it would violate the posix compatiblity.

For a test, can you set the values of the writer->reader to be "synchronous".

Things I'd try:  why use pipes for the network case? -- its prone to being less stable/having more problems than if you just use ssh for transfer.  I.e. spawn a process on client to run ssh and
connect client stdout to ssh-in, and pipe it directly through ssh.

Would simplify the logical design and if it works, you can focus on getting pipes fixed -- if you still
want to -- or just use ssh-direct.

Um... did I hear you say python?  Erk... might try perl or even c++!  (pullin' your leg)... :-)

FWIW, rsync can be set to use ssh via the RSYNC_RSH envvar and it works pretty reliably (though 
considerably slower than passing it in cleartext -- not something to worry about right now, though,
as first priority is getting it working, then focus on perf issues if needed.  

One other thing you could try -- **if the client can have access to the destination's file system
with samba, you could create a socket in the exported directory and the client could write
directly to the socket (w/a reader on the destination machine).  Again, if samba is unencrypted,
you'll get faster speeds.  I can regularly get greater than 100MB/s on a 1Gb or 10Gb link, when
writing to "/dev/null" on a remote machine.  I have gotten as high as 600MB/s on a 10Gb.  At 
that point the cpu is saturated --- which is why any encryption will slow that type of thing down.






@Astara with "pipe" i meant the inter-process pipe between borg and its subprocess ssh (on the client side) and sshd and its subprocess "borg serve" (on the server side). Sounds like you mean the same.

Not sure what you mean with "writer->reader to be synchronous" or how it would help here.
Nor what you mean with "ssh-direct".

If we have access to destination filesystem, we don't need to do client/server, we could also just directly write to the repo.
> Things I'd try: why use pipes for the network case?

These are the pipes from and to the respective SSH processes:

`borg client <-- pipe --> ssh <-- network --> sshd <-- pipe --> borg serve`.

> I have gotten as high as 600MB/s on a 10Gb. At
that point the cpu is saturated --- which is why any encryption will slow that type of thing down.

Depends on the encryption; modern CPUs do AEAD at a couple GB/s per core (eg. AES-128/256-GCM or even Chacha20-Poly1305).

I don't know if SMB can do AES (edit: the newest version deployed by Windows 10 does AES-GCM); SSH can and does use these efficient primitives; Chapoly is the default in recent versions.

Resolving patch:
diff --git a/borg/remote.py b/borg/remote.py
--- a/borg/remote.py
+++ b/borg/remote.py
@@ -7,6 +7,7 @@
 import sys
 import tempfile
 import textwrap
+import time
 from subprocess import Popen, PIPE
 
 from . import __version__
@@ -28,6 +29,23 @@
 MAX_INFLIGHT = 100
 
 
+def os_write(fd, data):
+    """os.write wrapper so we do not lose data for partial writes."""
+    # This is happening frequently on cygwin due to its small pipe buffer size of only 64kiB
+    # and also due to its different blocking pipe behaviour compared to Linux/*BSD.
+    # Neither Linux nor *BSD ever do partial writes on blocking pipes, unless interrupted by a
+    # signal, in which case serve() would terminate.
+    amount = remaining = len(data)
+    while remaining:
+        count = os.write(fd, data)
+        remaining -= count
+        if not remaining:
+            break
+        data = data[count:]
+        time.sleep(count * 1e-09)
+    return amount
+
+
 class ConnectionClosed(Error):
     """Connection closed by remote host"""
 
@@ -106,7 +124,7 @@ def serve(self):
                     if self.repository is not None:
                         self.repository.close()
                     else:
-                        os.write(stderr_fd, "Borg {}: Got connection close before repository was opened.\n"
+                        os_write(stderr_fd, "Borg {}: Got connection close before repository was opened.\n"
                                  .format(__version__).encode())
                     return
                 unpacker.feed(data)
@@ -133,9 +151,9 @@ def serve(self):
                             logging.exception('Borg %s: exception in RPC call:', __version__)
                             logging.error(sysinfo())
                         exc = "Remote Exception (see remote log for the traceback)"
-                        os.write(stdout_fd, msgpack.packb((1, msgid, e.__class__.__name__, exc)))
+                        os_write(stdout_fd, msgpack.packb((1, msgid, e.__class__.__name__, exc)))
                     else:
-                        os.write(stdout_fd, msgpack.packb((1, msgid, None, res)))
+                        os_write(stdout_fd, msgpack.packb((1, msgid, None, res)))
             if es:
                 self.repository.close()
                 return


Test patch: