Repository: Netflix/metaflow
Pull request: Pipe conda & Includefile setup logs to stderr

Problem statement:
IncludeFile and @conda logs should go to the stderr
Flows with **IncludeFile** generate info logs in the stdout.  They break a generated json when used with "step-functions create --only-json".  Prohibits to use the output with other tools (e.g. jq) through the shell pipelines.

Steps to reproduce:
```
$ python 02-statistics/stats.py --no-pylint step-functions create --only-json >tmp/out
```
And then check the generated file:
```
$ head -5 /tmp/out
2020-11-06 15:27:26.329 Including file 02-statistics/movies.csv of size 191KB 
2020-11-06 15:27:26.580 File persisted at s3://bucket/sysroot/data/MovieStatsFlow/9e3a38eb45be19331cf19b394d8191f54a06a281
{
    "StartAt": "start",
    "States": {
```

The same with the @conda decorator:
```
python 04-playlist-plus/playlist.py --environment=conda step-functions create --only-json >/tmp/out
```
With @conda it's even worse since a user doesn't see the message "Bootstrapping conda environment..." and just an empty screen doing nothing few minutes because the log message went to the stdout:
```
$ head -5 /tmp/out
2020-11-09 13:08:00.481 Bootstrapping conda environment...(this could take a few minutes)
{
    "StartAt": "start",
    "States": {
        "start": {
```

Hints from issue discussion:
Thanks for the issue. We will fix this promptly.

Commit message: Pipe conda & Includefile setup logs to stderr

Patch to review:
diff --git a/metaflow/cli.py b/metaflow/cli.py
index a6ea2b84cb7..35fb54091b3 100644
--- a/metaflow/cli.py
+++ b/metaflow/cli.py
@@ -90,15 +90,21 @@ def echo_always(line, **kwargs):
         click.secho(ERASE_TO_EOL, **kwargs)
 
 
-def logger(body='', system_msg=False, head='', bad=False, timestamp=True):
+def logger(body='',
+           system_msg=False,
+           head='',
+           bad=False,
+           timestamp=True,
+           err=False):
     if timestamp:
         tstamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
-        click.secho(tstamp + ' ', fg=LOGGER_TIMESTAMP, nl=False)
+        click.secho(tstamp + ' ', fg=LOGGER_TIMESTAMP, nl=False, err=err)
     if head:
-        click.secho(head, fg=LOGGER_COLOR, nl=False)
+        click.secho(head, fg=LOGGER_COLOR, nl=False, err=err)
     click.secho(body,
                 bold=system_msg,
-                fg=LOGGER_BAD_COLOR if bad else None)
+                fg=LOGGER_BAD_COLOR if bad else None,
+                err=err)
 
 
 @click.group()
diff --git a/metaflow/includefile.py b/metaflow/includefile.py
index 62b870334ab..fb43a01f837 100644
--- a/metaflow/includefile.py
+++ b/metaflow/includefile.py
@@ -309,7 +309,8 @@ def store(self, flow_name, path, is_text, encoding, logger):
         else:
             extra = ''
         logger(
-            'Including file %s of size %d%s %s' % (path, sz, unit[pos], extra))
+            'Including file %s of size %d%s %s' % (path, sz, unit[pos], extra),
+            err=True)
         try:
             cur_obj = TransformableObject(io.open(path, mode='rb').read())
         except IOError:
@@ -327,7 +328,7 @@ def store(self, flow_name, path, is_text, encoding, logger):
         buf.seek(0)
         with self._client_class() as client:
             url = client.put(path, buf.getvalue(), overwrite=False)
-            logger('File persisted at %s' % url)
+            logger('File persisted at %s' % url, err=True)
             return Uploader.encode_url(Uploader.file_type, url, is_text=is_text, encoding=encoding)
 
     def load(self, value):
diff --git a/metaflow/plugins/conda/conda_environment.py b/metaflow/plugins/conda/conda_environment.py
index 0388ac7c90e..2679967f488 100644
--- a/metaflow/plugins/conda/conda_environment.py
+++ b/metaflow/plugins/conda/conda_environment.py
@@ -20,7 +20,8 @@ def __init__(self, flow):
 
     def init_environment(self, logger):
         # Print a message for now
-        logger("Bootstrapping conda environment...(this could take a few minutes)")
+        logger("Bootstrapping conda environment..." +
+            "(this could take a few minutes)", err=True)
 
     def decospecs(self):
         # Apply conda decorator to all steps


Code review comments:
diff_hunk: @@ -90,15 +90,21 @@ def echo_always(line, **kwargs):
         click.secho(ERASE_TO_EOL, **kwargs)
 
 
-def logger(body='', system_msg=False, head='', bad=False, timestamp=True):
+def logger(body='',
+           system_msg=False,
+           head='',
+           bad=False,
+           timestamp=True,
+           err=False):
path: metaflow/cli.py
text: @author:
I am leaning towards setting the default value of `err` to True. Thoughts, @user1 ?