Repository: ansible/ansible

Issue:
tasks can not work with ansible_connection=lxc using python3
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
When using python3 and  ansible_connection=lxc,  tasks failed with an exeception. 

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
lxc connection plugin

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/toshi/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.5 (default, May 11 2018, 04:00:52) [GCC 8.1.0]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
set `remote_tmp = /tmp/`

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
Host: Arch linux
Lxc target: Debian Buster

I use latest python3-lxc module (https://github.com/lxc/python3-lxc).
I installed it by `python3 setup.py`

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
command:
```bash
$ sudo ansible-playbook -vvvv -i hosts rep.yml
```

inventory:
```yaml
target ansible_connection=lxc
```

playbook:
```
- hosts: target
  gather_facts: false
  tasks:
    - shell: echo a
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

Tasks should be success.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
ansible-playbook 2.5.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 3.6.5 (default, May 11 2018, 04:00:52) [GCC 8.1.0]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed /home/toshi/hosts inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3.6/site-packages/ansible/plugins/callback/default.py
PLAYBOOK: rep.yml *********************************************************************************************************************************************************************************************************************************************
1 plays in rep.yml
PLAY [target] *************************************************************************************************************************************************************************************************************************************************
META: ran handlers
TASK [shell] **************************************************************************************************************************************************************************************************************************************************
task path: /home/toshi/rep.yml:5
Using module file /usr/lib/python3.6/site-packages/ansible/modules/commands/command.py
<target> THIS IS A LOCAL LXC DIR
<target> EXEC [b'/bin/sh', '-c', b'/bin/sh -c \'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1528011515.7671123-231383058746331 `" && echo ansible-tmp-1528011515.7671123-231383058746331="` echo /tmp/ansible-tmp-1528011515.7671123-231383058746331 `" ) && sleep 0\'']
fatal: [target]: UNREACHABLE! => {
    "changed": false,
    "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /tmp/ansible-tmp-1528011515.7671123-231383058746331 `\" && echo ansible-tmp-1528011515.7671123-231383058746331=\"` echo /tmp/ansible-tmp-1528011515.7671123-231383058746331 `\" ), exited with result 65280, stderr output: Traceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/lxc/__init__.py\", line 457, in attach_run_command\nTypeError: argument 1 must be str, not bytes\n",
    "unreachable": true
}
        to retry, use: --limit @/home/toshi/rep.retry
PLAY RECAP ****************************************************************************************************************************************************************************************************************************************************
target                     : ok=0    changed=0    unreachable=1    failed=0
```

#### Possibly fix
I modified lxc connection plugin on my local machine, and it works fine.
The attached diff is my quick fix.
I think it only works on python3, It would break python2 (not tried)

```python
--- lxc.py      2018-06-03 16:30:51.632998463 +0900
+++ lxc.py.new  2018-06-03 16:37:53.177583642 +0900
@@ -116,8 +116,8 @@
         ''' run a command on the chroot '''
         super(Connection, self).exec_command(cmd, in_data=in_data, sudoable=sudoable)
-        executable = to_bytes(self._play_context.executable, errors='surrogate_or_strict')
-        local_cmd = [executable, '-c', to_bytes(cmd, errors='surrogate_or_strict')]
+        executable =self._play_context.executable
+        local_cmd = [executable, '-c', cmd]
         read_stdout, write_stdout = None, None
         read_stderr, write_stderr = None, None
```
tasks can not work with ansible_connection=lxc using python3
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
When using python3 and  ansible_connection=lxc,  tasks failed with an exeception. 

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
lxc connection plugin

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/toshi/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.5 (default, May 11 2018, 04:00:52) [GCC 8.1.0]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
set `remote_tmp = /tmp/`

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
Host: Arch linux
Lxc target: Debian Buster

I use latest python3-lxc module (https://github.com/lxc/python3-lxc).
I installed it by `python3 setup.py`

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
command:
```bash
$ sudo ansible-playbook -vvvv -i hosts rep.yml
```

inventory:
```yaml
target ansible_connection=lxc
```

playbook:
```
- hosts: target
  gather_facts: false
  tasks:
    - shell: echo a
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

Tasks should be success.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
ansible-playbook 2.5.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 3.6.5 (default, May 11 2018, 04:00:52) [GCC 8.1.0]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed /home/toshi/hosts inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3.6/site-packages/ansible/plugins/callback/default.py
PLAYBOOK: rep.yml *********************************************************************************************************************************************************************************************************************************************
1 plays in rep.yml
PLAY [target] *************************************************************************************************************************************************************************************************************************************************
META: ran handlers
TASK [shell] **************************************************************************************************************************************************************************************************************************************************
task path: /home/toshi/rep.yml:5
Using module file /usr/lib/python3.6/site-packages/ansible/modules/commands/command.py
<target> THIS IS A LOCAL LXC DIR
<target> EXEC [b'/bin/sh', '-c', b'/bin/sh -c \'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1528011515.7671123-231383058746331 `" && echo ansible-tmp-1528011515.7671123-231383058746331="` echo /tmp/ansible-tmp-1528011515.7671123-231383058746331 `" ) && sleep 0\'']
fatal: [target]: UNREACHABLE! => {
    "changed": false,
    "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /tmp/ansible-tmp-1528011515.7671123-231383058746331 `\" && echo ansible-tmp-1528011515.7671123-231383058746331=\"` echo /tmp/ansible-tmp-1528011515.7671123-231383058746331 `\" ), exited with result 65280, stderr output: Traceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/lxc/__init__.py\", line 457, in attach_run_command\nTypeError: argument 1 must be str, not bytes\n",
    "unreachable": true
}
        to retry, use: --limit @/home/toshi/rep.retry
PLAY RECAP ****************************************************************************************************************************************************************************************************************************************************
target                     : ok=0    changed=0    unreachable=1    failed=0
```

#### Possibly fix
I modified lxc connection plugin on my local machine, and it works fine.
The attached diff is my quick fix.
I think it only works on python3, It would break python2 (not tried)

```python
--- lxc.py      2018-06-03 16:30:51.632998463 +0900
+++ lxc.py.new  2018-06-03 16:37:53.177583642 +0900
@@ -116,8 +116,8 @@
         ''' run a command on the chroot '''
         super(Connection, self).exec_command(cmd, in_data=in_data, sudoable=sudoable)
-        executable = to_bytes(self._play_context.executable, errors='surrogate_or_strict')
-        local_cmd = [executable, '-c', to_bytes(cmd, errors='surrogate_or_strict')]
+        executable =self._play_context.executable
+        local_cmd = [executable, '-c', cmd]
         read_stdout, write_stdout = None, None
         read_stderr, write_stderr = None, None
```


Discussion:
Files identified in the description:
* [lib/ansible/plugins/connection/lxc.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/lxc.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Looks like the lxc API takes native string types instead of bytes types.  That's probably a bad design on their part (as they won't be able to handle all cases.. like non-utf8 filenames when operating under a utf8 locale).  However, that's they're issue, not ours.

The proper fix in our code for this would be to change the to_bytes() calls into to_native() calls.  And add a comment that says that we're to_native because lxc's API requires native strings.  (Usually, converting to native strings in controller side code is suspect so we should comment when we do so).

Would you like to open a pull request with that change?
Thanks for your explanation. I'll try to send a PR.
Files identified in the description:
* [lib/ansible/plugins/connection/lxc.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/lxc.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Looks like the lxc API takes native string types instead of bytes types.  That's probably a bad design on their part (as they won't be able to handle all cases.. like non-utf8 filenames when operating under a utf8 locale).  However, that's they're issue, not ours.

The proper fix in our code for this would be to change the to_bytes() calls into to_native() calls.  And add a comment that says that we're to_native because lxc's API requires native strings.  (Usually, converting to native strings in controller side code is suspect so we should comment when we do so).

Would you like to open a pull request with that change?
Thanks for your explanation. I'll try to send a PR.

Resolving patch:
diff --git a/lib/ansible/plugins/connection/lxc.py b/lib/ansible/plugins/connection/lxc.py
--- a/lib/ansible/plugins/connection/lxc.py
+++ b/lib/ansible/plugins/connection/lxc.py
@@ -45,7 +45,7 @@
 
 from ansible import constants as C
 from ansible import errors
-from ansible.module_utils._text import to_bytes
+from ansible.module_utils._text import to_bytes, to_native
 from ansible.plugins.connection import ConnectionBase
 
 
@@ -116,8 +116,9 @@ def exec_command(self, cmd, in_data=None, sudoable=False):
         ''' run a command on the chroot '''
         super(Connection, self).exec_command(cmd, in_data=in_data, sudoable=sudoable)
 
-        executable = to_bytes(self._play_context.executable, errors='surrogate_or_strict')
-        local_cmd = [executable, '-c', to_bytes(cmd, errors='surrogate_or_strict')]
+        # python2-lxc needs bytes. python3-lxc needs text.
+        executable = to_native(self._play_context.executable, errors='surrogate_or_strict')
+        local_cmd = [executable, '-c', to_native(cmd, errors='surrogate_or_strict')]
 
         read_stdout, write_stdout = None, None
         read_stderr, write_stderr = None, None


Test patch: