Repository: ansible/ansible

Issue:
eos_facts: intermittent failure in populate_memory: TypeError: string indices must be integers, not str
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
Use ansible to gather eos fact from DCS-7060CX-32S running EOS-4.18.4.2F. Observed following intermittent failure:
```
The full traceback is:
Traceback (most recent call last):
File "/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py", line 384, in <module>
main()
File "/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py", line 372, in main
inst.populate()
File "/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py", line 203, in populate
self.facts.update(self.populate_memory())
File "/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py", line 217, in populate_memory
memfree_mb=int(values['memFree']) / 1024,
TypeError: string indices must be integers, not str
```
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
eos_facts

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.2.0
  config file = /aos/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.14 (default, Feb 22 2018, 22:05:16) [GCC 6.2.1 20160822]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->

ANSIBLE_SSH_RETRIES(/aos/ansible.cfg) = 5
DEFAULT_FORKS(/aos/ansible.cfg) = 50
DEFAULT_GATHER_SUBSET(/aos/ansible.cfg) = !all,network
DEFAULT_GATHER_TIMEOUT(/aos/ansible.cfg) = 60
DEFAULT_HOST_LIST(/aos/ansible.cfg) = [u'/aos/hosts']
DEFAULT_REMOTE_TMP(/aos/ansible.cfg) = /tmp
DEFAULT_TIMEOUT(/aos/ansible.cfg) = 30
DISPLAY_SKIPPED_HOSTS(/aos/ansible.cfg) = False
HOST_KEY_CHECKING(/aos/ansible.cfg) = False
RETRY_FILES_ENABLED(/aos/ansible.cfg) = False

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->

OS hosting Ansible: Alpine Linux v3.5.2
OS I am managing: Arista EOS-4.18.4.2F

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

Simply run playbook against the said EOS switch. The problem is intermittent. 

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: "[EOS] Retrieving EOS facts"
  eos_facts:
    provider: "{{ eos_provider }}"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

EOS facts being populated.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Task failed.
<!--- Paste verbatim command output between quotes below -->
```
TASK [get_aos_device_facts : [EOS] Retrieving EOS facts] ***********************ansible.plugins.callback.apstra_operation_status:DEBUG:task start eos_facts get_aos_device_facts : [EOS] Retrieving EOS facts get_aos_device_facts

task path: /aos/roles/get_aos_device_facts/tasks/get_aos_facts_eos.yml:4
Tuesday 05 June 2018  00:33:55 +0000 (0:00:00.055)       0:01:54.747 **********
<192.168.120.16> using connection plugin network_cli
Using module file /usr/lib/python2.7/site-packages/ansible/modules/network/eos/eos_facts.py
<192.168.120.16> ESTABLISH LOCAL CONNECTION FOR USER: root
<192.168.120.16> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1528158837.69-45367331166540 `" && echo ansible-tmp-1528158837.69-45367331166540="` echo /tmp/ansible-tmp-1528158837.69-45367331166540 `" ) && sleep 0'
<192.168.120.16> PUT /tmp/tmpyyloSW TO /tmp/ansible-tmp-1528158837.69-45367331166540/eos_facts.py
<192.168.120.16> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-tmp-1528158837.69-45367331166540/ /tmp/ansible-tmp-1528158837.69-45367331166540/eos_facts.py && sleep 0'
<192.168.120.16> EXEC /bin/sh -c '/usr/bin/python /tmp/ansible-tmp-1528158837.69-45367331166540/eos_facts.py; rm -rf "/tmp/ansible-tmp-1528158837.69-45367331166540/" > /dev/null 2>&1 && sleep 0'ansible.plugins.callback.apstra_operation_status:DEBUG:task (u'get_aos_device_facts : [EOS] Retrieving EOS facts',) failed on 192.168.120.16

The full traceback is:
Traceback (most recent call last):
File "/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py", line 384, in <module>
main()
File "/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py", line 372, in main
inst.populate()
File "/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py", line 203, in populate
self.facts.update(self.populate_memory())
File "/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py", line 217, in populate_memory
memfree_mb=int(values['memFree']) / 1024,
TypeError: string indices must be integers, not str
fatal: [192.168.120.16]: FAILED! => {
"changed": false,
"module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py\", line 384, in <module>\n    main()\n  File \"/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py\", line 372, in main\n    inst.populate()\n  File \"/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py\", line 203, in populate\n    self.facts.update(self.populate_memory())\n  File \"/tmp/ansible_hh4_Fh/ansible_module_eos_facts.py\", line 217, in populate_memory\n    memfree_mb=int(values['memFree']) / 1024,\nTypeError: string indices must be integers, not str\n",vansible.plugins.callback.apstra_operation_status:DEBUG:ansible.plugins.callback.apstra_operation_status: final task started
ansible.plugins.callback.apstra_operation_status:DEBUG:ansible.plugins.callback.apstra_operation_status: overflow buffer 0
ansible.plugins.callback.apstra_operation_status:DEBUG:ansible.plugins.callback.apstra_operation_status: max overflow buffer 0
ansible.plugins.callback.apstra_operation_status:DEBUG:ansible.plugins.callback.apstra_operation_status: amount written (bytes) 25529
Nansible.plugins.callback.apstra_operation_status:DEBUG:failure 192.168.120.16
ansible.plugins.callback.apstra_operation_status:DEBUG:flush time started 2018-06-05 00:33:59.149606
ansible.plugins.callback.apstra_operation_status:DEBUG:flush attempt 0 length 0
ansible.plugins.callback.apstra_operation_status:DEBUG:flush time complete 2018-06-05 00:33:59.149869 duration 0:00:00.000263
ansible.plugins.callback.apstra_operation_status:DEBUG:ansible.plugins.callback.apstra_operation_status: final task done
M
"module_stdout": "",
"msg": "MODULE FAILURE",
"rc": 0
}
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/network/eos/eos_facts.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/eos/eos_facts.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @privateip @trishnaguha
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@DanCool Thank you for logging your observation but I am unable to reproduce the intermittent behavior in ansible 2.4, can you please re-verify the same in ansible 2.5 or later version if you are still facing the respective issue. 
Thanks very much for looking into it. I don't have a way to consistently reproduce it even with 2.4. It seems EOS "show version | json" returned a string (error?) instead of json by looking at the code and exception. If that is the case, do you think 2.5 will help? We don't have a concrete plan to upgrade to 2.5 yet. So I am not sure when I will be able to provide the test result.

Resolving patch:
diff --git a/lib/ansible/module_utils/network/eos/eos.py b/lib/ansible/module_utils/network/eos/eos.py
--- a/lib/ansible/module_utils/network/eos/eos.py
+++ b/lib/ansible/module_utils/network/eos/eos.py
@@ -176,7 +176,12 @@ def run_commands(self, commands, check_rc=True):
                 prompt = None
                 answer = None
 
-            out = connection.get(command, prompt, answer)
+            try:
+                out = connection.get(command, prompt, answer)
+            except ConnectionError as exc:
+                if check_rc:
+                    raise
+                out = getattr(exc, 'err', exc)
             out = to_text(out, errors='surrogate_or_strict')
 
             try:
@@ -338,7 +343,7 @@ def send_request(self, commands, output='text'):
 
         return response
 
-    def run_commands(self, commands):
+    def run_commands(self, commands, check_rc=True):
         """Runs list of commands on remote device and returns results
         """
         output = None
@@ -489,9 +494,9 @@ def get_config(module, flags=None):
     return conn.get_config(flags)
 
 
-def run_commands(module, commands):
+def run_commands(module, commands, check_rc=True):
     conn = get_connection(module)
-    return conn.run_commands(to_command(module, commands))
+    return conn.run_commands(to_command(module, commands), check_rc)
 
 
 def load_config(module, config, commit=False, replace=False):
diff --git a/lib/ansible/modules/network/eos/eos_facts.py b/lib/ansible/modules/network/eos/eos_facts.py
--- a/lib/ansible/modules/network/eos/eos_facts.py
+++ b/lib/ansible/modules/network/eos/eos_facts.py
@@ -153,7 +153,7 @@ def __init__(self, module):
         self.responses = None
 
     def populate(self):
-        self.responses = run_commands(self.module, list(self.COMMANDS))
+        self.responses = run_commands(self.module, list(self.COMMANDS), check_rc=False)
 
 
 class Default(FactsBase):
@@ -258,7 +258,8 @@ def populate(self):
         self.facts['interfaces'] = self.populate_interfaces(data)
 
         data = self.responses[1]
-        self.facts['neighbors'] = self.populate_neighbors(data['lldpNeighbors'])
+        if data:
+            self.facts['neighbors'] = self.populate_neighbors(data['lldpNeighbors'])
 
     def populate_interfaces(self, data):
         facts = dict()


Test patch: