Repository: ansible/ansible
Pull request: RoleRequirement - Include stderr in the error message
##### SUMMARY
Fixes #41336

##### ISSUE TYPE
- Bugfix Pull Request

##### COMPONENT NAME
lib/ansible/playbook/role/requirement.py


Problem statement:
ansible-galaxy hides scm errors
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
When downloading roles from SCM repos using `ansible-galaxy -r requirements.yml`, the output from the SCM commands is dumped to `/dev/null`. This makes it very difficult to debug issues, particularly in non-interactive environments where all you have is a log to go by. It would be better to capture the SCM command output and display it (perhaps only when ANSIBLE_VERBOSITY > 1 to minimize noise).

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
ansible-galaxy
##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->

```
 ansible-galaxy 2.5.0
config file = None
configured module search path = ['/var/lib/jenkins/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /opt/ansible/2.5.0.0/lib/python3.5/site-packages/ansible
executable location = /opt/ansible/2.5.0.0/bin/ansible-galaxy
python version = 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
N/A

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
N/A

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

`requirements.yml`
```yaml
- role: myrole
  src: <scm url>
  scm: git
```

It is assumed that SCM url is either inaccessible or otherwise incorrect in a way that ansible-galaxy cannot detect

`ansible-galaxy install -r requirements.yml`

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
`
[WARNING]: - myrole was NOT installed successfully: - command git clone <scm_url> myrole failed in directory /some/tmp/dir (rc=128). Command output follows:
<command output>
`
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
[WARNING]: - myrole was NOT installed successfully: - command git clone
14:38:15     baseline/cis-level-1: <scm url> myrole failed in directory /tmp/tmpglah_hf5 (rc=128)
```

Or something like that.

Hints from issue discussion:
Files identified in the description:
* [lib/ansible/galaxy](https://github.com/ansible/ansible/blob/devel/lib/ansible/galaxy)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
+1 
cc @alikins
i believe this was already fixed https://github.com/ansible/ansible/pull/36788
nope, it's still useless:

https://github.com/ansible/ansible/blob/3a3465c47df0d4a680bdc7490790567d920f3919/lib/ansible/playbook/role/requirement.py#L152

Commit message: Don't try to concatenate str and bytes

Patch to review:
diff --git a/lib/ansible/playbook/role/requirement.py b/lib/ansible/playbook/role/requirement.py
index 8e974f16848abd..62741680109bdf 100644
--- a/lib/ansible/playbook/role/requirement.py
+++ b/lib/ansible/playbook/role/requirement.py
@@ -145,11 +145,11 @@ def run_scm_cmd(cmd, tempdir):
             except Exception as e:
                 ran = " ".join(cmd)
                 display.debug("ran %s:" % ran)
-                display.debug("\tstdout: " + stdout)
-                display.debug("\tstderr: " + stderr)
+                display.debug("\tstdout: " + to_native(stdout))
+                display.debug("\tstderr: " + to_native(stderr))
                 raise AnsibleError("when executing %s: %s" % (ran, to_native(e)))
             if popen.returncode != 0:
-                raise AnsibleError("- command %s failed in directory %s (rc=%s)" % (' '.join(cmd), tempdir, popen.returncode))
+                raise AnsibleError("- command %s failed in directory %s (rc=%s) - %s" % (' '.join(cmd), tempdir, popen.returncode, to_native(stderr)))
 
         if scm not in ['hg', 'git']:
             raise AnsibleError("- scm %s is not currently supported" % scm)


Code review comments:
diff_hunk: @@ -145,11 +145,11 @@ def run_scm_cmd(cmd, tempdir):
             except Exception as e:
                 ran = " ".join(cmd)
                 display.debug("ran %s:" % ran)
-                display.debug("\tstdout: " + stdout)
-                display.debug("\tstderr: " + stderr)
+                display.debug("\tstdout: " + to_native(stdout))
path: lib/ansible/playbook/role/requirement.py
text: @user1:
```suggestion
                display.debug("\tstdout: " + to_text(stdout))
```

diff_hunk: @@ -145,11 +145,11 @@ def run_scm_cmd(cmd, tempdir):
             except Exception as e:
                 ran = " ".join(cmd)
                 display.debug("ran %s:" % ran)
-                display.debug("\tstdout: " + stdout)
-                display.debug("\tstderr: " + stderr)
+                display.debug("\tstdout: " + to_native(stdout))
+                display.debug("\tstderr: " + to_native(stderr))
path: lib/ansible/playbook/role/requirement.py
text: @user1:
```suggestion
                display.debug("\tstderr: " + to_text(stderr))
```