Repository: ansible/ansible

Issue:
Ansible facts override special variables
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
Certain scenarios can result in internal ansible facts being overridden, this was originally fixed in v2.4.x however this bug (previously reported: https://github.com/ansible/ansible/issues/16935) has been reintroduced on version 2.5.0+ and still exists on the devel branch.

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
gather_facts


##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.7.0.dev0 (devel 1737b7be3e) last updated 2018/06/18 15:00:52 (GMT +100)
  config file = None
  configured module search path = [u'/home/sam/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/sam/Work/src/github.com/ansible/ansible/lib/ansible
  executable location = /home/sam/pythonEnv/bin/ansible
  python version = 2.7.12+ (default, Sep  1 2016, 20:27:38) [GCC 6.2.0 20160927]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
Observed on RHEL, Centos 7, Debian Jessie and Ubuntu 16.04 

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

Create a network interface with a name that conflicts with an ansible internal variable, in this case I'm using a veth pair (with one end called "user" and other end "password") but it could be a tap interface or a vlan interface.
```
ip link add user type veth peer name password
```

Run a playbook with two plays:

<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- name: Do the actual task
  hosts: all
  tasks:
    - debug:
        msg: "{{ansible_user}}"
    - name: install vim
      package:
        name: vim
        state: present
      become: true
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

```
PLAY [Do the actual task] ************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************
 [WARNING]: Removed restricted key from module data: ansible_password = {u'macaddress ... ther'}

 [WARNING]: Removed restricted key from module data: ansible_user = {u'macaddress ... ther'}

ok: [localhost]

TASK [debug] *************************************************************************************************************************
ok: [localhost] => {
    "msg": "sam"
}

TASK [install vim] *******************************************************************************************************************
ok: [localhost]

PLAY RECAP ***************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   

```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```

PLAY [Do the actual task] ************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************
 [WARNING]: Removed restricted key from module data: ansible_password = {u'macaddress ... ther'}

 [WARNING]: Removed restricted key from module data: ansible_user = {u'macaddress ... ther'}

ok: [localhost]

TASK [debug] *************************************************************************************************************************
ok: [localhost] => {
    "msg": {
        "active": false, 
        "device": "user", 
        "macaddress": "9a:67:85:d7:9e:e7", 
        "mtu": 1500, 
        "promisc": false, 
        "type": "ether"
    }
}

TASK [install vim] *******************************************************************************************************************
fatal: [localhost]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: command-line line 0: garbage at end of line; \"u'9a:67:85:d7:9e:e7',\".\r\n", "unreachable": true}
	to retry, use: --limit @/home/sam/Work/sandbox/playbook.retry

PLAY RECAP ***************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=1    failed=0   
```



Discussion:
I have resolve this locally by making this change:

```
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
index e2adba0d73..c8e6c7bedb 100644
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -624,7 +624,8 @@ class TaskExecutor:
                 else:
                     vars_copy.update(namespace_facts(result['ansible_facts']))
                     if C.INJECT_FACTS_AS_VARS:
-                        vars_copy.update(clean_facts(result['ansible_facts']))
+                        result['ansible_facts'] = clean_facts(result['ansible_facts'])
+                        vars_copy.update(result['ansible_facts'])
 
             # set the failed property if it was missing.
             if 'failed' not in result:
@@ -685,7 +686,8 @@ class TaskExecutor:
             else:
                 variables.update(namespace_facts(result['ansible_facts']))
                 if C.INJECT_FACTS_AS_VARS:
-                    variables.update(clean_facts(result['ansible_facts']))
+                    result['ansible_facts'] = clean_facts(result['ansible_facts'])
+                    variables.update(result['ansible_facts'])
 
         # save the notification target in the result, if it was specified, as
         # this task may be running in a loop in which case the notification
```

However I'm ensure if this logic should move to a more common location for example:
https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/strategy/__init__.py#L606
@Tehsmash the code change you provided seems like a sane fix, if you'd like to open a pull request I'll review ... or I can open the PR and credit you in the commit message. Either way, thank you! :)

Resolving patch:
diff --git a/lib/ansible/constants.py b/lib/ansible/constants.py
--- a/lib/ansible/constants.py
+++ b/lib/ansible/constants.py
@@ -22,7 +22,7 @@ def _warning(msg):
     try:
         from __main__ import display
         display.warning(msg)
-    except:
+    except Exception:
         import sys
         sys.stderr.write(' [WARNING] %s\n' % (msg))
 
@@ -32,7 +32,7 @@ def _deprecated(msg, version='2.8'):
     try:
         from __main__ import display
         display.deprecated(msg, version=version)
-    except:
+    except Exception:
         import sys
         sys.stderr.write(' [DEPRECATED] %s, to be removed in %s\n' % (msg, version))
 
@@ -53,7 +53,7 @@ def get_config(parser, section, key, env_var, default_value, value_type=None, ex
     if value is None:
         try:
             value = get_ini_config_value(parser, {'key': key, 'section': section})
-        except:
+        except Exception:
             pass
     if value is None:
         value = default_value
@@ -123,6 +123,10 @@ def set_constant(name, value, export=vars()):
 # object. The dictionary values are tuples, to account for aliases
 # in variable names.
 
+COMMON_CONNECTION_VARS = frozenset(set(('ansible_connection', 'ansbile_host', 'ansible_user', 'ansible_shell_executable',
+                                        'ansible_port', 'ansible_pipelining', 'ansible_password', 'ansible_timeout',
+                                        'ansible_shell_type', 'ansible_module_compression', 'ansible_private_key_file')))
+
 MAGIC_VARIABLE_MAPPING = dict(
 
     # base
@@ -193,7 +197,7 @@ def set_constant(name, value, export=vars()):
                 value = literal_eval(value)
             except ValueError:
                 pass  # not a python data structure
-        except:
+        except Exception:
             pass  # not templatable
 
         value = ensure_type(value, setting.type)
diff --git a/lib/ansible/vars/clean.py b/lib/ansible/vars/clean.py
--- a/lib/ansible/vars/clean.py
+++ b/lib/ansible/vars/clean.py
@@ -63,8 +63,13 @@ def clean_facts(facts):
     fact_keys = set(data.keys())
     # first we add all of our magic variable names to the set of
     # keys we want to remove from facts
+    # NOTE: these will eventually disappear in favor of others below
     for magic_var in C.MAGIC_VARIABLE_MAPPING:
         remove_keys.update(fact_keys.intersection(C.MAGIC_VARIABLE_MAPPING[magic_var]))
+
+    # remove common connection vars
+    remove_keys.update(fact_keys.intersection(C.COMMON_CONNECTION_VARS))
+
     # next we remove any connection plugin specific vars
     for conn_path in connection_loader.all(path_only=True):
         try:
@@ -72,7 +77,7 @@ def clean_facts(facts):
             re_key = re.compile('^ansible_%s_' % conn_name)
             for fact_key in fact_keys:
                 # most lightweight VM or container tech creates devices with this pattern, this avoids filtering them out
-                if re_key.match(fact_key) and not fact_key.endswith(('_bridge', '_gwbridge')):
+                if (re_key.match(fact_key) and not fact_key.endswith(('_bridge', '_gwbridge'))) or re_key.startswith('ansible_become_'):
                     remove_keys.add(fact_key)
         except AttributeError:
             pass
diff --git a/lib/ansible/vars/manager.py b/lib/ansible/vars/manager.py
--- a/lib/ansible/vars/manager.py
+++ b/lib/ansible/vars/manager.py
@@ -43,7 +43,7 @@
 from ansible.utils.listify import listify_lookup_plugin_terms
 from ansible.utils.vars import combine_vars
 from ansible.utils.unsafe_proxy import wrap_var
-from ansible.vars.clean import namespace_facts
+from ansible.vars.clean import namespace_facts, clean_facts
 
 try:
     from __main__ import display
@@ -313,7 +313,7 @@ def plugins_by_groups():
 
                 # push facts to main namespace
                 if C.INJECT_FACTS_AS_VARS:
-                    all_vars = combine_vars(all_vars, wrap_var(facts))
+                    all_vars = combine_vars(all_vars, wrap_var(clean_facts(facts)))
                 else:
                     # always 'promote' ansible_local
                     all_vars = combine_vars(all_vars, wrap_var({'ansible_local': facts.get('ansible_local', {})}))


Test patch: