Repository: sonic-net/sonic-mgmt

Issue:
Docker shell module failed in the latest sonic-mgmt
<!--
If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below. If you fail to provide this
information within 7 days, we cannot debug your issue and will close it. We
will, however, reopen it if you later provide the information.

For more information about reporting issues, see
https://github.com/Azure/SONiC/wiki#report-issues

---------------------------------------------------
GENERAL SUPPORT INFORMATION
---------------------------------------------------

The GitHub issue tracker is for bug reports and feature requests.
General support can be found at the following locations:

- SONiC Support Forums - https://groups.google.com/forum/#!forum/sonicproject

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

**Description**
The task with sell type docker always fails on the latest sonic-mgmt, example:
```
- name: Gather information from lldp
  lldp:
  vars:
    ansible_shell_type: docker
    ansible_python_interpreter: docker exec -i lldp python
```

<!--
Briefly describe the problem you are having in a few paragraphs.
-->

**Steps to reproduce the issue:**
1. run dip_sip or lag_2 CT 
2.
3.

**Describe the results you received:**
```
TASK [test : Gathering peer VM information from lldp] **************************
task path: /var/user/jenkins/bfn-sonic-mgmt/ansible/roles/test/tasks/lag_2.yml:26
Thursday 28 November 2019  10:28:47 +0000 (0:00:00.263)       0:00:26.753 ***** 
The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 145, in run
    res = self._execute()
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 612, in _execute
    self._set_connection_options(variables, templar)
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 1012, in _set_connection_options
    self._set_plugin_options('shell', final_vars, templar, task_keys)
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 978, in _set_plugin_options
    plugin.set_options(task_keys=task_keys, var_options=options)
  File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/shell/__init__.py", line 70, in set_options
    env = self.get_option('environment')
  File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/__init__.py", line 60, in get_option
    raise KeyError(to_native(e))
KeyError: 'Requested entry (plugin_type: shell plugin: docker setting: environment ) was not defined in configuration.'

fatal: [cab18-2-dut]: FAILED! => {
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}

```

**Describe the results you expected:**


**Additional information you deem important:**
<!--
software/ASIC/Hardware Flatform version and info
-->
    **Output of `show version`:**

    ```
SONiC Software Version: SONiC.HEAD.488-dirty-20191127.082217
Distribution: Debian 9.11
Kernel: 4.9.0-9-2-amd64
Build commit: 7622a30d
Build date: Wed Nov 27 11:15:51 UTC 2019
Built by: johnar@jenkins-worker-11

    ```

    **Attach debug file `sudo generate_dump`:**

    ```
    (paste your output here)
    ```



Discussion:
I have tried to update the python docker module to the latest as we still have docker-py=1.6.0 when ansible requires docker>= 1.7.0 but this did not help me.
https://docs.ansible.com/ansible/latest/scenario_guides/guide_docker.html#requirements
As lldp module is run on sonic host with custom python interpreter ```ansible_python_interpreter: docker exec -i lldp python``` I've tried to commend ```ansible_shell_type: docker``` and this helps me.
@nazariig I see you are the author of this [line ```ansible_shell_type: docker```](https://github.com/Azure/sonic-mgmt/blame/e8892cf3587afe38e51f2df5630ce23ff80c9a55/ansible/roles/test/tasks/dip_sip.yml#L26) 
Could you comment?
> 
> 
> @nazariig I see you are the author of this [line `ansible_shell_type: docker`](https://github.com/Azure/sonic-mgmt/blame/e8892cf3587afe38e51f2df5630ce23ff80c9a55/ansible/roles/test/tasks/dip_sip.yml#L26)
> Could you comment?

@msosyak This module allows you to execute commands directly in docker on the remote host (inventory)
plugin: https://github.com/Azure/sonic-mgmt/blob/master/ansible/shell_plugins/docker.py
FYI: https://docs.ansible.com/ansible/latest/plugins/shell.html
@msosyak please check this: https://github.com/Azure/sonic-mgmt/pull/1213 and https://github.com/Azure/sonic-mgmt/pull/1225
@qiluo-msft  is the original author of docker shell plugin, i think i might break the plugin when upgrading 2.8.7. I do not know if it is still necessary. if you are able to run the lldp module successfully, then we no longer need  ```ansible_shell_type: docker```
I did some test. for lldp module, you can remove both lines.

```
    ansible_shell_type: docker
    ansible_python_interpreter: docker exec -i lldp python
```

I think we have lldpctl in sonic base image.

Resolving patch:
diff --git a/ansible/shell_plugins/docker.py b/ansible/shell_plugins/docker.py
--- a/ansible/shell_plugins/docker.py
+++ b/ansible/shell_plugins/docker.py
@@ -1,6 +1,17 @@
 from __future__ import (absolute_import, division)
 __metaclass__ = type
 
+DOCUMENTATION = '''
+name: docker
+plugin_type: shell
+short_description: "docker shell plugin"
+version_added: historical
+description:
+  - This module allows you to execute commands directly in docker on the remote host
+extends_documentation_fragment:
+  - shell_common
+'''
+
 import os
 import re
 import pipes
@@ -29,7 +40,24 @@ def join_path(self, *args):
 
         return super(ShellModule, self).join_path(*args)
 
-    def build_module_command(self, env_string, shebang, cmd, arg_path=None, rm_tmp=None):
+    def remove(self, path, recurse=False):
+        argv = self.get_option('ansible_python_interpreter').split()
+        assert(argv[0] == 'docker')
+        assert(argv[1] == 'exec')
+        opts, args = getopt.getopt(argv[2:], 'i')
+        self.container_name = args[0]
+
+        remove_files_on_host_cmd = super(ShellModule, self).remove(path, recurse)
+
+        cmd = remove_files_on_host_cmd + "; docker exec -i "
+        cmd += self.container_name + " rm -f "
+        if recurse:
+            cmd += '-r '
+        cmd += " ".join(self.dtemps)
+
+        return cmd
+
+    def build_module_command(self, env_string, shebang, cmd, arg_path=None):
         # assert(self.container_name)
         argv = shlex.split(shebang.replace("#!", ""))
         assert(argv[0] == 'docker')
@@ -51,13 +79,7 @@ def build_module_command(self, env_string, shebang, cmd, arg_path=None, rm_tmp=N
         pre = ''.join('docker exec {1} mkdir -p {0}; docker cp {0}/. {1}:{0}; '
             .format(dtemp, self.container_name) for dtemp in self.dtemps)
 
-        if rm_tmp:
-            post = ''.join('docker exec {1} rm -rf {0}; '
-                .format(dtemp, self.container_name) for dtemp in self.dtemps)
-        else:
-            post = ''
-
-        return pre + super(ShellModule, self).build_module_command('', shebang_env, cmd, arg_path, rm_tmp) + '; ' + post
+        return pre + super(ShellModule, self).build_module_command('', shebang_env, cmd, arg_path)
 
     def checksum(self, path, python_interp):
         """


Test patch: