Repository: iterative/dvc

Issue:
SSH: Fail to push binary data
## Description
I get an error when trying to push generated model files to my remote over ssh.

## General steps to reproduce:
* Init dvc repository
* Add an ssh remote
* Train the model using `dvc run`
* DVC push
The error I get is : `Error: Failed to push data to the cloud: sequence item 0: expected str instance, bytes found`

## More details
* First, I am just beginning to use DVC, so I may have missed a step.
* The output of `dvc run...` is a directory, which contains several files. Some are text files, but others are binary files.




Discussion:
Hi @midemarc !

What version of dvc are you using(`dvc --version`)? Also, could you please try running `dvc push` with `--verbose` flag and show us the output?

Thanks,
Ruslan
Hi @efiop, I use dvc 0.18.15.
Here is the full output of the push : 
```(.env) ❯ dvc push -v
Preparing to push data to ssh://marcq@myserver.com:/data/legalia
[######                        ] 20% Collecting informationDebug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
[##############################] 100% Collecting information
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/50/db147ec5d6cd5fa391006858aecf7f' to 'myserver.com//data/legalia/50/db147ec5d6cd5fa391006858aecf7f'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/9e/6f5573f5865b9d15976bffbb426ce9' to 'myserver.com//data/legalia/9e/6f5573f5865b9d15976bffbb426ce9'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/3d/fc6d3d5610b9f881fe8594c0a2c070' to 'myserver.com//data/legalia/3d/fc6d3d5610b9f881fe8594c0a2c070'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/dc/f534628569ee3dd8d0e2308ea251fc' to 'myserver.com//data/legalia/dc/f534628569ee3dd8d0e2308ea251fc'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/07/9225f11226fd5edcd20db53fd0c322' to 'myserver.com//data/legalia/07/9225f11226fd5edcd20db53fd0c322'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/cd/101a915c405fefd0a07ffcb1d0a202' to 'myserver.com//data/legalia/cd/101a915c405fefd0a07ffcb1d0a202'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/63/cecd443da6719ab1c9afe208c390db' to 'myserver.com//data/legalia/63/cecd443da6719ab1c9afe208c390db'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/17/be35239b627fc66bde3846d53abf2c' to 'myserver.com//data/legalia/17/be35239b627fc66bde3846d53abf2c'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/41/955b350dc6d962b4829a207d178191' to 'myserver.com//data/legalia/41/955b350dc6d962b4829a207d178191'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/40/a585bc24f6e39a4336d945b5b35e44' to 'myserver.com//data/legalia/40/a585bc24f6e39a4336d945b5b35e44'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/3b/0adcc9ad63e924a5a4c18502cc6b44.dir' to 'myserver.com//data/legalia/3b/0adcc9ad63e924a5a4c18502cc6b44.dir'
Error: Traceback (most recent call last):
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/command/data_sync.py", line 41, in do_run
    all_branches=self.args.all_branches)
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/project.py", line 420, in push
    show_checksums=show_checksums)
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/data_cloud.py", line 108, in push
    show_checksums=show_checksums)
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/remote/local.py", line 667, in push
    f.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/remote/ssh.py", line 374, in upload
    self._exec(ssh, 'mkdir -p {}'.format(dname))
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/remote/ssh.py", line 147, in _exec
    err = ''.join(stderr_chunks)
TypeError: sequence item 0: expected str instance, bytes found

Error: Failed to push data to the cloud: sequence item 0: expected str instance, bytes found
```
Thank you for your help !
Looks like we are improperly joining the stderr for when the ssh fails. I will send a patch ASAP and it is going to be released with v0.19 in the nearest days. But the core issue I think is that there is too much concurrency for your ssh server to handle. Could you please try limit the number of connections with the `-j` flag? I.e. could you please try running `dvc push -j 4 -v` ?

Thanks,
Ruslan
It's still the same, with 4 or even 1 job(s) : 
```
(.env) ❯ dvc push -j 4 -v
Preparing to push data to ssh://marcq@myserver.com:/data/legalia
[######                        ] 20% Collecting informationDebug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
[##############################] 100% Collecting information
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Establishing ssh connection with 'myserver.com' through port '22' as user 'marcq'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/41/955b350dc6d962b4829a207d178191' to 'myserver.com//data/legalia/41/955b350dc6d962b4829a207d178191'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/9e/6f5573f5865b9d15976bffbb426ce9' to 'myserver.com//data/legalia/9e/6f5573f5865b9d15976bffbb426ce9'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/63/cecd443da6719ab1c9afe208c390db' to 'myserver.com//data/legalia/63/cecd443da6719ab1c9afe208c390db'
Debug: Uploading '/home/marc/dev/rasa_test/.dvc/cache/cd/101a915c405fefd0a07ffcb1d0a202' to 'myserver.com//data/legalia/cd/101a915c405fefd0a07ffcb1d0a202'
Error: Traceback (most recent call last):
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/command/data_sync.py", line 41, in do_run
    all_branches=self.args.all_branches)
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/project.py", line 420, in push
    show_checksums=show_checksums)
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/data_cloud.py", line 108, in push
    show_checksums=show_checksums)
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/remote/local.py", line 667, in push
    f.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/remote/ssh.py", line 374, in upload
    self._exec(ssh, 'mkdir -p {}'.format(dname))
  File "/home/marc/dev/rasa_test/.env/lib/python3.6/site-packages/dvc/remote/ssh.py", line 147, in _exec
    err = ''.join(stderr_chunks)
TypeError: sequence item 0: expected str instance, bytes found

Error: Failed to push data to the cloud: sequence item 0: expected str instance, bytes found
```

Resolving patch:
diff --git a/dvc/remote/ssh.py b/dvc/remote/ssh.py
--- a/dvc/remote/ssh.py
+++ b/dvc/remote/ssh.py
@@ -152,7 +152,7 @@ def _exec(self, ssh, cmd):
         stderr.close()
 
         if stdout.channel.recv_exit_status() != 0:
-            err = ''.join(stderr_chunks)
+            err = b''.join(stderr_chunks).decode('utf-8')
             msg = 'SSH command \'{}\' failed: {}'.format(cmd, err)
             raise DvcException(msg)
 


Test patch: