Repository: horovod/horovod

Issue:
Horovod throws UnicodeDecodeError when using tqdm
**Environment:**
1. Framework: PyTorch
2. Framework version: 1.2.0
3. Horovod version: 0.19.5
4. MPI version: None
5. CUDA version: 10.0
6. NCCL version: 2.7.8.1
7. Python version: 3.6.12
8. Spark / PySpark version: None
9. OS and version: Linux 4.14.105-1-tlinux3-0010
10. GCC version: 5.2.0
11. CMake version: 2.8.12.2

**What happened?**
When using `horovod` together with `tqdm`, `horovod` throws `UnicodeDecodeError` when `tqdm` is progressing the bar display:

```
Tue Oct 13 10:54:25 2020[0]<stderr>:[10/13/2020 10:54:25 - INFO - __main__ -   start running validation...
Tue Oct 13 10:54:25 2020[0]<stderr>:  0%|          | 0/1327 [00:00<?, ?it/s]
Tue Oct 13 10:55:56 2020[0]<stderr>:
Exception in thread Thread-17:derr>: 65%|██████▌   | 866/1327 [01:17<00:47,  9.80it/s]
Traceback (most recent call last):
  File "/data/anaconda3/envs/vcr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/data/anaconda3/envs/vcr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/runner/common/util/safe_shell_exec.py", line 104, in forward_stream
    text = text.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 999: unexpected end of data

Tue Oct 13 10:58:56 2020[3]<stderr>:[2020-10-13 10:58:56.365049: E /dockerdata/app/tmp/pip-install-998q9kn4/horovod/horovod/common/operations.cc:525] Horovod background loop uncaught exception: [/dockerdata/app/tmp/pip-install-998q9kn4/horovod/third_party/compatible_gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 30000ms for recv operation to complete
Tue Oct 13 10:58:56 2020[2]<stderr>:[2020-10-13 10:58:56.366492: E /dockerdata/app/tmp/pip-install-998q9kn4/horovod/horovod/common/operations.cc:525] Horovod background loop uncaught exception: [/dockerdata/app/tmp/pip-install-998q9kn4/horovod/third_party/compatible_gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 30000ms for recv operation to complete
Tue Oct 13 10:58:56 2020[1]<stderr>:[2020-10-13 10:58:56.366674: E /dockerdata/app/tmp/pip-install-998q9kn4/horovod/horovod/common/operations.cc:525] Horovod background loop uncaught exception: [/dockerdata/app/tmp/pip-install-998q9kn4/horovod/third_party/compatible_gloo/gloo/transport/tcp/unbound_buffer.cc:136] Timed out waiting 30000ms for send operation to complete
Tue Oct 13 10:58:56 2020[3]<stderr>:Traceback (most recent call last):
Tue Oct 13 10:58:56 2020[1]<stderr>:Traceback (most recent call last):
Tue Oct 13 10:58:56 2020[3]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 622, in synchronize
Tue Oct 13 10:58:56 2020[2]<stderr>:Traceback (most recent call last):
Tue Oct 13 10:58:56 2020[3]<stderr>:    mpi_lib.horovod_torch_wait_and_clear(handle)
Tue Oct 13 10:58:56 2020[2]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 622, in synchronize
Tue Oct 13 10:58:56 2020[3]<stderr>:RuntimeError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
Tue Oct 13 10:58:56 2020[1]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 622, in synchronize
Tue Oct 13 10:58:56 2020[3]<stderr>:
Tue Oct 13 10:58:56 2020[1]<stderr>:    mpi_lib.horovod_torch_wait_and_clear(handle)
Tue Oct 13 10:58:56 2020[2]<stderr>:    mpi_lib.horovod_torch_wait_and_clear(handle)
Tue Oct 13 10:58:57 2020[3]<stderr>:During handling of the above exception, another exception occurred:
Tue Oct 13 10:58:57 2020[2]<stderr>:RuntimeError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
Tue Oct 13 10:58:57 2020[1]<stderr>:RuntimeError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
Tue Oct 13 10:58:57 2020[3]<stderr>:
Tue Oct 13 10:58:57 2020[1]<stderr>:
Tue Oct 13 10:58:57 2020[2]<stderr>:
Tue Oct 13 10:58:57 2020[3]<stderr>:Traceback (most recent call last):
Tue Oct 13 10:58:57 2020[2]<stderr>:During handling of the above exception, another exception occurred:
Tue Oct 13 10:58:57 2020[1]<stderr>:During handling of the above exception, another exception occurred:
Tue Oct 13 10:58:57 2020[3]<stderr>:  File "train_vcr.py", line 332, in <module>
Tue Oct 13 10:58:57 2020[1]<stderr>:
Tue Oct 13 10:58:57 2020[3]<stderr>:    main(parse_cmd_args(TrainingOpts))
Tue Oct 13 10:58:57 2020[1]<stderr>:Traceback (most recent call last):
Tue Oct 13 10:58:57 2020[3]<stderr>:  File "train_vcr.py", line 260, in main
Tue Oct 13 10:58:57 2020[1]<stderr>:  File "train_vcr.py", line 332, in <module>
Tue Oct 13 10:58:57 2020[3]<stderr>:    val_log = validate(model, val_dataloader)
Tue Oct 13 10:58:57 2020[1]<stderr>:    main(parse_cmd_args(TrainingOpts))
Tue Oct 13 10:58:57 2020[3]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
Tue Oct 13 10:58:57 2020[1]<stderr>:  File "train_vcr.py", line 260, in main
Tue Oct 13 10:58:57 2020[3]<stderr>:    return func(*args, **kwargs)
Tue Oct 13 10:58:57 2020[1]<stderr>:    val_log = validate(model, val_dataloader)
Tue Oct 13 10:58:58 2020[3]<stderr>:  File "train_vcr.py", line 310, in validate
Tue Oct 13 10:58:58 2020[1]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
Tue Oct 13 10:58:57 2020[2]<stderr>:
Tue Oct 13 10:58:58 2020[3]<stderr>:    qa_loss = sum(all_gather_list(qa_loss))
Tue Oct 13 10:58:58 2020[1]<stderr>:    return func(*args, **kwargs)
Tue Oct 13 10:58:58 2020[2]<stderr>:Traceback (most recent call last):
Tue Oct 13 10:58:58 2020[1]<stderr>:  File "train_vcr.py", line 310, in validate
Tue Oct 13 10:58:58 2020[2]<stderr>:  File "train_vcr.py", line 332, in <module>
Tue Oct 13 10:58:58 2020[3]<stderr>:  File "/data/cdp_algo_ceph_ssd/users/haoyanhuo/vcr/utils/distributed.py", line 233, in all_gather_list
Tue Oct 13 10:58:58 2020[2]<stderr>:    main(parse_cmd_args(TrainingOpts))
Tue Oct 13 10:58:58 2020[3]<stderr>:    max_size = hvd.allgather(torch.tensor([enc_size]).cuda()).max().item()
Tue Oct 13 10:58:58 2020[1]<stderr>:    qa_loss = sum(all_gather_list(qa_loss))
Tue Oct 13 10:58:58 2020[2]<stderr>:  File "train_vcr.py", line 260, in main
Tue Oct 13 10:58:58 2020[3]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 365, in allgather
Tue Oct 13 10:58:58 2020[2]<stderr>:    val_log = validate(model, val_dataloader)
Tue Oct 13 10:58:58 2020[3]<stderr>:    return HorovodAllgather.apply(tensor, name)
Tue Oct 13 10:58:58 2020[1]<stderr>:  File "/data/cdp_algo_ceph_ssd/users/haoyanhuo/vcr/utils/distributed.py", line 233, in all_gather_list
Tue Oct 13 10:58:58 2020[2]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
Tue Oct 13 10:58:58 2020[1]<stderr>:    max_size = hvd.allgather(torch.tensor([enc_size]).cuda()).max().item()
Tue Oct 13 10:58:58 2020[3]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 328, in forward
Tue Oct 13 10:58:59 2020[2]<stderr>:    return func(*args, **kwargs)
Tue Oct 13 10:58:59 2020[3]<stderr>:    return synchronize(handle)
Tue Oct 13 10:58:59 2020[2]<stderr>:  File "train_vcr.py", line 310, in validate
Tue Oct 13 10:58:59 2020[3]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 626, in synchronize
Tue Oct 13 10:58:59 2020[2]<stderr>:    qa_loss = sum(all_gather_list(qa_loss))
Tue Oct 13 10:58:59 2020[3]<stderr>:    raise HorovodInternalError(e)
Tue Oct 13 10:58:59 2020[2]<stderr>:  File "/data/cdp_algo_ceph_ssd/users/haoyanhuo/vcr/utils/distributed.py", line 233, in all_gather_list
Tue Oct 13 10:58:59 2020[3]<stderr>:horovod.common.exceptions.HorovodInternalError: Horovod has been shut down. This was causedby an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finishedexecution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
Tue Oct 13 10:58:59 2020[1]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 365, in allgather
Tue Oct 13 10:58:59 2020[2]<stderr>:    max_size = hvd.allgather(torch.tensor([enc_size]).cuda()).max().item()
Tue Oct 13 10:58:59 2020[2]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 365, in allgather
Tue Oct 13 10:58:59 2020[1]<stderr>:    return HorovodAllgather.apply(tensor, name)
Tue Oct 13 10:58:59 2020[1]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 328, in forward
Tue Oct 13 10:58:59 2020[2]<stderr>:    return HorovodAllgather.apply(tensor, name)
Tue Oct 13 10:58:59 2020[2]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 328, in forward
Tue Oct 13 10:58:59 2020[1]<stderr>:    return synchronize(handle)
Tue Oct 13 10:58:59 2020[1]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 626, in synchronize
Tue Oct 13 10:58:59 2020[2]<stderr>:    return synchronize(handle)
Tue Oct 13 10:58:59 2020[1]<stderr>:    raise HorovodInternalError(e)
Tue Oct 13 10:59:00 2020[2]<stderr>:  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/torch/mpi_ops.py", line 626, in synchronize
Tue Oct 13 10:59:00 2020[1]<stderr>:horovod.common.exceptions.HorovodInternalError: Horovod has been shut down. This was causedby an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finishedexecution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
Tue Oct 13 10:59:00 2020[2]<stderr>:    raise HorovodInternalError(e)
Tue Oct 13 10:59:00 2020[2]<stderr>:horovod.common.exceptions.HorovodInternalError: Horovod has been shut down. This was causedby an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finishedexecution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message.
```

**Reasons for this issue**
This is due to a truncated (since `horovod` only captures 1000 bytes) UTF8 code. For example, the above error was due to a truncated black box character `0xe2 0x96 0x88` used by `tqdm` to indicate progress. The following is an example of the bytes produced by `tqdm`. Pay attention to bytes `0x00059960-0x00059970`:
```
...
00059930  30 5d 3c 73 74 64 65 72  72 3e 3a 0d 54 75 65 20  |0]<stderr>:.Tue |
00059940  4f 63 74 20 31 33 20 31  30 3a 35 35 3a 35 35 20  |Oct 13 10:55:55 |
00059950  32 30 32 30 5b 30 5d 3c  73 74 64 65 72 72 3e 3a  |2020[0]<stderr>:|
00059960  20 36 35 25 7c e2 96 88  e2 96 88 e2 96 88 e2 96  | 65%|...........|
00059970  88 e2 96 88 e2 96 88 e2  96 8d 20 20 20 7c 20 38  |..........   | 8|
00059980  35 38 2f 31 33 32 37 20  5b 30 31 3a 31 37 3c 30  |58/1327 [01:17<0|
00059990  30 3a 34 34 2c 20 31 30  2e 34 33 69 74 2f 73 5d  |0:44, 10.43it/s]|
000599a0  1b 5b 41 1b 5b 41 0a 54  75 65 20 4f 63 74 20 31  |.[A.[A.Tue Oct 1|
000599b0  33 20 31 30 3a 35 35 3a  35 35 20 32 30 32 30 5b  |3 10:55:55 2020[|
000599c0  30 5d 3c 73 74 64 65 72  72 3e 3a 0a 54 75 65 20  |0]<stderr>:.Tue |
...
```

I think `horovod` needs a better way of handling program outputs, either pipe them in binary or delay the decoding of partially captured UTF8 codes.



Discussion:
how to solve this bug?
> how to solve this bug?

A quick fix: `tqdm(ascii=True)` to eliminate UTF8 outputs.

The better way: Replace all `str`s in `forward_stream` with `bytes`.
> > how to solve this bug?
> 
> A quick fix: `tqdm(ascii=True)` to eliminate UTF8 outputs.
> 
> The better way: Replace all `str`s in `forward_stream` with `bytes`.

thanks!
Thanks for raising this issue @hhaoyan.  Do you have a small repro script I can run to demonstrate the error?
> Thanks for raising this issue @hhaoyan. Do you have a small repro script I can run to demonstrate the error?

Here is a simple script that reproduces this error:

```python
from horovod import torch as hvd
import time

hvd.init()
while True:
    print(b'\xe2\x96\x88'.decode('utf8')*334)
    time.sleep(1)
```

From terminal, run:

```bash
$ horovodrun -n 2 pythontest.py
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/data/anaconda3/envs/vcr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/data/anaconda3/envs/vcr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/run/common/util/safe_shell_exec.py", line 98, in forward_stream
    text = text.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 999: unexpected end of data
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/data/anaconda3/envs/vcr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/data/anaconda3/envs/vcr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/data/anaconda3/envs/vcr/lib/python3.6/site-packages/horovod/run/common/util/safe_shell_exec.py", line 98, in forward_stream
    text = text.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 999: unexpected end of data
```
> how to solve this bug?

Alternative would be not to truncate on 1000 bytes but 1000 characters. Where is this 1000 truncation happening?
> > how to solve this bug?
> 
> Alternative would be not to truncate on 1000 bytes but 1000 characters. Where is this 1000 truncation happening?

https://github.com/horovod/horovod/blob/master/horovod/runner/common/util/safe_shell_exec.py#L99

If subprocess throws some illegal UTF8 bytes, then I guess there is no way to truncate in characters?

I temporarily fixed it by using bytes IO for all pipes, stdout/stderr and it seems to be working for me.
I think the simplest fix is to use `text.decode` with `errors = 'ignore'`, which makes it ignore incomplete / invalid UTF8 characters: https://docs.python.org/3.6/howto/unicode.html#the-string-type

https://github.com/horovod/horovod/blob/master/horovod/runner/common/util/safe_shell_exec.py#L104 would then read:

    text = text.decode('utf-8', errors='ignore')

Good suggestion, @EnricoMi.  Do you have the bandwidth to put together a PR?
Sure, give me a day or two, I would also add a unit test for that.
After looking at the `forward_stream` method a bit closer, using `errors='ignore'` is not the right thing to do here. We are not truncating the stream at 1000 bytes, but we are chunking it into 1000 bytes chunks to line buffer it. We would lose the character at the chunk boundary. I will rework the line buffering.
@tgaddair: is it required for `forward_stream` to work with both, byte streams and text streams? Given we are calling this from `execute` on pipe connections, we can guarantee it is one of them. Would simplify things.

```
        if not isinstance(text, str):
            text = text.decode('utf-8')
```

https://github.com/horovod/horovod/blob/master/horovod/runner/common/util/safe_shell_exec.py#L103
@EnricoMi it may not be necessary to decode the stream data, I do not remember why we needed to do that originally, but we can always decode on the other end of the stream when necessary.
We have to encode it in `forward_stream` to be able to prepend timestamp, prefix and index to each line:

    return '{time}[{rank}]<{prefix}>:{line}'.format(...)

I think this justifies expecting the stream to be a text stream.
Waiting for #2388 to land which simplifies unit test assertions without timestamps.
> We have to encode it in `forward_stream` to be able to prepend timestamp, prefix and index to each line:
> 
> ```
> return '{time}[{rank}]<{prefix}>:{line}'.format(...)
> ```
> 
> I think this justifies expecting the stream to be a text stream.

Maybe the prefix strings can be formatted and encoded into UTF8, then we can simply join two byte strings?
That is a good idea, that would then only require the source stream to be split on new lines, no matter if being a byte or text stream. If a byte stream, prepend each line as described by you, if a text stream, prepend with the prefix as a string. Will try that.

Resolving patch:
diff --git a/horovod/runner/common/util/safe_shell_exec.py b/horovod/runner/common/util/safe_shell_exec.py
--- a/horovod/runner/common/util/safe_shell_exec.py
+++ b/horovod/runner/common/util/safe_shell_exec.py
@@ -13,9 +13,9 @@
 # limitations under the License.
 # ==============================================================================
 
+import codecs
 import multiprocessing
 import os
-import psutil
 import re
 import signal
 import subprocess
@@ -23,6 +23,8 @@
 import threading
 import time
 
+import psutil
+
 from horovod.runner.util.threads import in_thread, on_event
 
 GRACEFUL_TERMINATION_TIME_S = 5
@@ -78,45 +80,69 @@ def terminate_executor_shell_and_children(pid):
             pass
 
 
-def forward_stream(src_stream, dst_stream, prefix, index, prefix_output_with_timestamp):
-    def prepend_context(line, rank, prefix):
+def prefix_connection(src_connection, dst_stream, prefix, index, prefix_output_with_timestamp):
+    """
+    Prefixes the given source connection with timestamp, a prefix and an index.
+    Each line of the source will be prefix in this format, if index and prefix are not None:
+        {time}[{index}]<{prefix}>:{line}
+    The dst_stream must be text streams.
+
+    :param src_connection: source pipe connection
+    :param dst_stream: destination text stream
+    :param prefix: prefix string
+    :param index: index value
+    :param prefix_output_with_timestamp: prefix lines in dst_stream with timestamp
+    :return: None
+    """
+    def get_context(rank, prefix):
         localtime = time.asctime(time.localtime(time.time())) if prefix_output_with_timestamp else ''
-        return '{time}[{rank}]<{prefix}>:{line}'.format(
+        return '{time}[{rank}]<{prefix}>:'.format(
             time=localtime,
             rank=str(rank),
-            prefix=prefix,
-            line=line
+            prefix=prefix
         )
 
     def write(text):
-        if index is not None:
-            text = prepend_context(text, index, prefix)
+        if index is not None and prefix is not None:
+            context = get_context(index, prefix)
+            dst_stream.write(context)
         dst_stream.write(text)
         dst_stream.flush()
 
+    # the incremental encoder allows us to decode chunks of utf8 bytes
+    # with utf8 characters spread across the boundary chunks
+    decoder = codecs.getincrementaldecoder('utf8')()
+
     line_buffer = ''
     while True:
-        text = os.read(src_stream.fileno(), 1000)
-        if text is None:
-            break
-
-        if not isinstance(text, str):
-            text = text.decode('utf-8')
-
-        if not text:
-            break
-
+        # read at most that many bytes, but do not wait until that many bytes become available
+        # waits for the first bytes so this is not a busy loop
+        buf = os.read(src_connection.fileno(), 1000)
+
+        # turn the bytes into string decoding them as utf8
+        # we need to use an incremental decoder as characters can span multiple bytes
+        # where the last character might not be completely in buf
+        # see https://github.com/horovod/horovod/issues/2367
+        text = decoder.decode(buf or b'', final=not buf)
+
+        # write line_buffer out when we reach an \n or \r
+        # the latter is used to update the current line (e.g. progress bar)
+        # which we want to flush out (and prefix) as soon as possible
         for line in re.split('([\r\n])', text):
             line_buffer += line
             if line == '\r' or line == '\n':
                 write(line_buffer)
                 line_buffer = ''
 
+        # not an empty text indicates EOF but an empty buf
+        if not buf:
+            break
+
     # flush the line buffer if it is not empty
     if len(line_buffer):
         write(line_buffer)
 
-    src_stream.close()
+    src_connection.close()
 
 
 def _exec_middleman(command, env, exit_event, stdout, stderr, rw):
@@ -134,7 +160,7 @@ def _exec_middleman(command, env, exit_event, stdout, stderr, rw):
                                       stdout=stdout_w, stderr=stderr_w)
 
     # we don't bother stopping the on_event thread, this process sys.exits soon
-    # so the on_event thread has to be a deamon thread
+    # so the on_event thread has to be a daemon thread
     on_event(exit_event, terminate_executor_shell_and_children, args=(executor_shell.pid,), daemon=True)
 
     def kill_executor_children_if_parent_dies():
@@ -159,7 +185,24 @@ def _create_event(ctx):
     return ctx.Event()
 
 
-def execute(command, env=None, stdout=None, stderr=None, index=None, events=None, prefix_output_with_timestamp=False):
+def execute(command, env=None, stdout=None, stderr=None, index=None, events=None,
+            prefix_output_with_timestamp=False):
+    """
+    Execute the given command and forward stdout and stderr of the command to the given
+    stdout and stderr text streams, or sys.stdout and sys.stderr, respectively, if None given.
+    Prefixes each line with index and timestamp if index is not None. The timestamp
+    can be disabled with prefix_output_with_timestamp set False.
+    The command will be terminated when any of the given events are set.
+
+    :param command: command to execute
+    :param env: environment variables to execute command with
+    :param stdout: stdout text stream, sys.stdout if None
+    :param stderr: stderr text stream, sys.stderr if None
+    :param index: index used to prepend text streams
+    :param events: events to terminate the command
+    :param prefix_output_with_timestamp: prepend text streams with timestamp if True
+    :return: command's exit code
+    """
     ctx = multiprocessing.get_context('spawn')
 
     # When this event is set, signal to middleman to terminate its children and exit.
@@ -195,8 +238,8 @@ def execute(command, env=None, stdout=None, stderr=None, index=None, events=None
     if stderr is None:
         stderr = sys.stderr
 
-    stdout_fwd = in_thread(target=forward_stream, args=(stdout_r, stdout, 'stdout', index, prefix_output_with_timestamp))
-    stderr_fwd = in_thread(target=forward_stream, args=(stderr_r, stderr, 'stderr', index, prefix_output_with_timestamp))
+    stdout_fwd = in_thread(target=prefix_connection, args=(stdout_r, stdout, 'stdout', index, prefix_output_with_timestamp))
+    stderr_fwd = in_thread(target=prefix_connection, args=(stderr_r, stderr, 'stderr', index, prefix_output_with_timestamp))
 
     # TODO: Currently this requires explicitly declaration of the events and signal handler to set
     #  the event (gloo_run.py:_launch_jobs()). Need to figure out a generalized way to hide this behind


Test patch:
diff --git a/test/single/test_run.py b/test/single/test_run.py
--- a/test/single/test_run.py
+++ b/test/single/test_run.py
@@ -16,6 +16,8 @@
 import copy
 import io
 import itertools
+import logging
+import multiprocessing
 import os
 import subprocess
 import sys
@@ -24,28 +26,26 @@
 import unittest
 import warnings
 
+import mock
 import psutil
 import pytest
-import mock
-
+from common import is_built, lsf_and_jsrun, override_args, override_env, temppath, delay, wait
 from mock import MagicMock
 
 import horovod
+from horovod.runner import _HorovodArgs
 from horovod.runner.common.util import config_parser, hosts, safe_shell_exec, secret, \
     settings as hvd_settings, timeout
-from horovod.runner import _HorovodArgs
 from horovod.runner.common.util.host_hash import _hash, host_hash
 from horovod.runner.common.util.hosts import SlotInfo, get_host_assignments, parse_hosts
 from horovod.runner.gloo_run import gloo_run
 from horovod.runner.js_run import js_run, generate_jsrun_rankfile
-from horovod.runner.mpi_run import _get_mpi_implementation, _get_mpi_implementation_flags,\
-    _LARGE_CLUSTER_THRESHOLD as large_cluster_threshold, mpi_available, mpi_run,\
-    _OMPI_IMPL, _SMPI_IMPL, _MPICH_IMPL, _UNKNOWN_IMPL, _MISSING_IMPL
 from horovod.runner.launch import gloo_built, parse_args, run_controller, _run
+from horovod.runner.mpi_run import _get_mpi_implementation, _get_mpi_implementation_flags, \
+    _LARGE_CLUSTER_THRESHOLD as large_cluster_threshold, mpi_available, mpi_run, \
+    _OMPI_IMPL, _SMPI_IMPL, _MPICH_IMPL, _UNKNOWN_IMPL, _MISSING_IMPL
 from horovod.runner.util.threads import in_thread, on_event
 
-from common import is_built, lsf_and_jsrun, override_args, override_env, temppath, delay, wait
-
 
 class RunTests(unittest.TestCase):
     """
@@ -343,6 +343,186 @@ def exception():
             on_event(event, fn, stop=None, daemon=False)
         fn.assert_not_called()
 
+    def test_prefix_connection(self):
+        string = 'first line\nsecond line\nmore lines\n'
+        self.do_test_prefix_connection(string, prefix='prefix', index=123,
+                                       expected='[123]<prefix>:first line\n'
+                                                '[123]<prefix>:second line\n'
+                                                '[123]<prefix>:more lines\n')
+
+    def test_prefix_connection_with_unicode(self):
+        string = '∀∁∂∃∄∅∆∇∈∉∊∋∌∍∎∏∐∑\n⌚⌛⛄✅…\n'
+        self.do_test_prefix_connection(string, prefix='prefix', index=123,
+                                       expected='[123]<prefix>:∀∁∂∃∄∅∆∇∈∉∊∋∌∍∎∏∐∑\n'
+                                                '[123]<prefix>:⌚⌛⛄✅…\n')
+
+    def test_prefix_connection_with_multibyte_unicode(self):
+        block = '█▓▒░▒▓'*1000
+        string = block + '\n' + block + '\n'
+        self.do_test_prefix_connection(string, prefix='prefix', index=123,
+                                       expected='[123]<prefix>:' + block + '\n' +
+                                                '[123]<prefix>:' + block + '\n')
+
+    def test_prefix_connection_with_timestamp(self):
+        string = 'first line\nsecond line\nmore lines\n'
+        self.do_test_prefix_connection_with_timestamp(
+            string, prefix='prefix', index=123,
+            expected='Mon Jan 20 12:00:01 2020[123]<prefix>:first line\n'
+                     'Mon Jan 20 12:00:02 2020[123]<prefix>:second line\n'
+                     'Mon Jan 20 12:00:03 2020[123]<prefix>:more lines\n'
+        )
+
+    def test_prefix_connection_with_timestamp_without_index(self):
+        string = 'first line\nsecond line\nmore lines\n'
+        self.do_test_prefix_connection_with_timestamp(string, prefix=None, index=None,
+                                                      expected=string)
+
+    def test_prefix_connection_without_trailing_newline(self):
+        string = 'first line\nsecond line\nmore lines'
+        self.do_test_prefix_connection(string, prefix='prefix', index=123,
+                                       expected='[123]<prefix>:first line\n'
+                                            '[123]<prefix>:second line\n'
+                                            '[123]<prefix>:more lines')
+
+    def test_prefix_connection_without_index(self):
+        string = 'first line\nsecond line\nmore lines\n'
+        self.do_test_prefix_connection(string, prefix='prefix', index=None, expected=string)
+
+    def test_prefix_connection_without_prefix(self):
+        string = 'first line\nsecond line\nmore lines\n'
+        self.do_test_prefix_connection(string, prefix=None, index=123, expected=string)
+
+    def test_prefix_connection_with_carriage_return(self):
+        string = 'first line\rfirst line again\nsecond line\n'
+        self.do_test_prefix_connection(string, prefix='prefix', index=123,
+                                       expected='[123]<prefix>:first line\r'
+                                            '[123]<prefix>:first line again\n'
+                                            '[123]<prefix>:second line\n')
+
+    def test_prefix_connection_with_carriage_return_without_index(self):
+        string = 'first line\rfirst line again\nsecond line\n'
+        self.do_test_prefix_connection(string, prefix='prefix', index=None, expected=string)
+
+    def test_prefix_connection_with_carriage_return_without_prefix(self):
+        string = 'first line\rfirst line again\nsecond line\n'
+        self.do_test_prefix_connection(string, prefix=None, index=123, expected=string)
+
+    def do_test_prefix_connection(self, string, prefix, index, expected, timestamp=False):
+        # create a Pipe Connection and populate it with string
+        (connection, w) = multiprocessing.get_context('spawn').Pipe()
+        with os.fdopen(w.fileno(), 'wt', encoding='utf8', newline='', closefd=False) as stream:
+            stream.write(string)
+        w.close()
+
+        dst = io.StringIO()
+        safe_shell_exec.prefix_connection(connection, dst, prefix=prefix, index=index,
+                                          prefix_output_with_timestamp=timestamp)
+        self.assertEqual(expected, dst.getvalue())
+
+    def do_test_prefix_connection_with_timestamp(self, string_or_connection, prefix, index, expected):
+        # control the time used to prepend the timestamp
+        class MockTime:
+            def __init__(self):
+                # start at 2020-01-20 12:00:00
+                self._time = 1579521600
+
+            def time(self, seconds):
+                from time import gmtime
+                self._time = self._time + 1
+                return gmtime(self._time)
+
+        with mock.patch('horovod.runner.common.util.safe_shell_exec.time.localtime',
+                        side_effect=MockTime().time):
+            self.do_test_prefix_connection(string_or_connection, prefix, index, expected, timestamp=True)
+
+    def test_prefix_connection_does_stream(self):
+        index = 123
+        prefix = 'prefix'
+        expected = '[123]<prefix>:first line\r' \
+                   '[123]<prefix>:first line again\n' \
+                   '[123]<prefix>:second line\n' \
+                   '[123]<prefix>:more lines'
+
+        timeout = 0.2
+        barrier = multiprocessing.Barrier(2, timeout=timeout)
+
+        def writer(write_connection):
+            def write(stream, text):
+                stream.write(text)
+                logging.info('wrote: {}'.format(text))
+                stream.flush()
+
+                # are we expecting the reader to read something
+                logging.info('waiting for reader')
+                if '\r' in text or '\n' in text:
+                    # yes, barrier should not timeout
+                    try:
+                        barrier.wait()
+                    except threading.BrokenBarrierError:
+                        self.fail('reader side should have read something from the stream')
+                else:
+                    # no, barrier should timeout
+                    try:
+                        barrier.wait()
+                        self.fail('reader side should not have read anything from the stream')
+                    except threading.BrokenBarrierError:
+                        logging.info('reader did not read')
+                        barrier.reset()
+                        pass
+                logging.info('continuing writing')
+
+            try:
+                with os.fdopen(write_connection.fileno(), 'wt', encoding='utf8', newline='', closefd=False) as stream:
+                    for text in ['first line\r',
+                                 'first ', 'line ', 'again\n',
+                                 'second line\nmore ', 'lines']:
+                        write(stream, text)
+            finally:
+                write_connection.close()
+
+        actual = []
+        do_read = True
+
+        def reader(read_connection):
+            try:
+                while do_read:
+                    text = os.read(read_connection.fileno(), 1000)
+                    if not text:
+                        break
+                    text = text.decode('utf8')
+                    actual.append(text)
+                    logging.info('read: {}'.format(text))
+                    logging.info('waiting for writer')
+                    try:
+                        barrier.wait()
+                    except threading.BrokenBarrierError as e:
+                        if do_read:
+                            raise e
+                        break
+                    logging.info('continuing reading')
+            finally:
+                read_connection.close()
+
+        # one thread writes into the w side of this pipe
+        # prefix_connection reads on the other end of this pipe
+        (connection, w) = multiprocessing.get_context('spawn').Pipe()
+        writer_thread = in_thread(writer, (w,))
+
+        # prefix_connection writes to the write side of this Pipe (opened as a text stream)
+        # another thread reads from the r side of this pipe
+        (r, dst_con) = multiprocessing.get_context('spawn').Pipe()
+        reader_thread = in_thread(reader, (r,))
+
+        with os.fdopen(dst_con.fileno(), 'wt', encoding='utf8', newline='', closefd=False) as dst:
+            safe_shell_exec.prefix_connection(connection, dst, prefix=prefix, index=index,
+                                              prefix_output_with_timestamp=False)
+
+        writer_thread.join(2*timeout)
+        do_read = False
+        reader_thread.join(2*timeout)
+
+        self.assertEqual(expected, ''.join(actual))
+
     def test_safe_shell_exec_captures_stdout(self):
         self.do_test_safe_shell_exec('echo hello', 0, 'hello\n', '')