Repository: ansible/molecule

Issue:
Ansible inventory SSH options for better testinfra support
Testinfra doesn't inherit all of the SSH options that Molecule uses during the converge phase which was causing testinfra to not be able to run because it couldn't SSH to the VM. It was prompting to accept the SSH fingerprint and would just hang. Adding SSH's StrictHostKeyChecking=no and UserKnownHostsFile=/dev/null to the ansible_inventory file allows testinfra to properly SSH to the VM and perform the tests.

Before the change:

```
Executing testinfra tests found in tests/.
============================= test session starts ==============================
platform darwin -- Python 2.7.11, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /Users/me/code/ansible/roles/foo, inifile: pytest.ini
plugins: testinfra-1.2.0
collected 4 items

tests/test_foo.py The authenticity of host '[127.0.0.1]:2200 ([127.0.0.1]:2200)' can't be established.
ECDSA key fingerprint is SHA256:uBXjhivauuYiPdv47uwCBxZNElAKY3MaPn5UCF7BEuw.
```

And at this point the command would hang forever.

After the change:

```
Executing testinfra tests found in tests/.
============================= test session starts ==============================
platform darwin -- Python 2.7.11, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /Users/me/code/ansible/roles/foo, inifile: pytest.ini
plugins: testinfra-1.2.0
collected 4 items

tests/test_foo.py ....

=========================== 4 passed in 3.08 seconds ===========================
```



Discussion:
How's that?

Not _just_ Denver, I live in Parker too! My profile just says Denver because non-locals often don't know about Parker.

Hah, crazy about Parker, it seems like I never encounter other tech types here.

Anyway, this LGTM. @retr0h ?

Although I wonder if we shouldn't piggyback on the same options that we're passing to SSH for `molecule login`?

https://github.com/metacloud/molecule/blob/master/molecule/conf/defaults.yml#L48-L51

Well first off, I'm not entirely sure how this bug is occurring.  We have over a hundred roles using molecule + testinfra, and none of them fail where testinfra is unable to ssh into the VM.  Rather than accepting this PR, I would rather understand what is different with your setup.

👎 

@retr0h I'm happy to troubleshoot but I think I'm using a fairly standard setup. I cannot put my whole module github yet (my company hasn't figured out their open source policies) but I'll share what I can.

I'm running OS X 10.11.4, VirtualBox 5.0.10, Vagrant 1.8.1, pyenv with Python 2.7.11 in a virtualenv.

My virtual env has: ansible==2.0.2.0, molecule==1.4.1, testinfra==1.2.0 plus, of course, all of their dependencies.

My molecule.yml has trusty64 and centos7 with a couple triggers to precreate/install/configure things that the role under test should fix/delete/etc.

``` yaml
---
vagrant:
  raw_config_args:
    - "ssh.insert_key = false"

  platforms:
    - name: trusty64
      box: trusty64
      box_url: https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/14.04/providers/virtualbox.box
      triggers:
        - trigger: after
          action: up
          cmd: run_remote 'redacted'
    - name: centos7
      box: centos/7
      box_url: https://vagrantcloud.com/centos/boxes/7/versions/1603.01/providers/virtualbox.box
      triggers:
        - trigger: after
          action: up
          cmd: run_remote 'redacted'

  providers:
    - name: virtualbox
      type: virtualbox

  instances:
    - name: foo
      interfaces:
        - network_name: private_network
          type: dhcp
          auto_config: true
```

My tests are using the standard testinfra modules.

My ssh config does not have any Host sections that would match the Vagrant VM.

Running molecule create, converge, idempotence, login, etc. all work just fine. It's only the tests that fail. If I run the tests without my changes it hangs waiting for me to accept the RSA fingerprint. However, since stdin is not under my control I cannot accept it. Nor should I since that would pollute my known_hosts.

Here's the testinfra command line that molecule is running when it hangs:
`/Users/mklich/.virtualenvs/ansible/bin/testinfra tests --connection=ansible --sudo --ansible-inventory=.molecule/ansible_inventory`

Here's the `.molecule/ansible_inventory` file when it hangs:

```
foo ansible_ssh_host=127.0.0.1 ansible_ssh_port=2200 ansible_ssh_private_key_file=/Users/mklich/.vagrant.d/insecure_private_key ansible_ssh_user=vagrant
```

Given that command and that ansible inventory there is now way for testinfra to automatically connect/accept the RSA fingerprint. It doesn't use the settings in the ansible.cfg (which molecule creates).

As I said, all of the molecule commands work including converge and login. However, they both run SSH using the settings from `raw_ssh_args` as @abrown-sg pointed out.

@elementalvoid Out of curiosity, does the problem persist if you take your ~/.ssh/config out of the picture?

Yes, it absolutely still happens. My ssh config is very much not custom. It has `ForwardX11 yes` and `ForwardAgent yes` plus some very specific host matches.

In your follow up to the above question (via email, it isn't in the comment thread) you said you had `StrictHostKeyChecking no` in your ssh config under the `Host *` section. This is why yours works and mine doesn't. This will auto accept the RSA fingerprint that I'm getting prompted to accept. I'm okay disabling strict host key checking on an automation system where I have explicit knowledge of what's happening and when it's happening (i.e. I invoked some command or a Jenkins build is doing so).  Generally speaking, it is not a good idea to always disable it. I have disabled it for my Jenkins slaves but I cannot add it to my user-wide ssh config.

I think the suggestion to append the `raw_ssh_args` is probably the right approach.

Yeah, sorry, I deleted that thinking it wasn't relevant. Because I removed it and didn't have the issue you're encountering.

I'm wondering if I added it at some point because of a similar issue and forgot to remove it. Not the kind of setting I'd normally roll with.

Haven't forgotten you here, I just want to do a little testing in a clean environment to understand exactly what's up. We're wrapping up a project and should have more cycles to dedicate to molecule soon.

Alright, finally got a chance to pull this into a clean system (ubuntu 14.04) without any sort of ssh config and duplicated the issue.

```
abrown@hype  ~/demo-01  molecule verify

Executing testinfra tests found in tests/.
============================= test session starts ==============================
platform linux2 -- Python 2.7.6, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: /home/abrown/demo-01/tests, inifile:
plugins: testinfra-1.2.0
collected 1 items

tests/test_default.py The authenticity of host '[127.0.0.1]:2222 ([127.0.0.1]:2222)' can't be established.
ECDSA key fingerprint is f7:6c:a9:15:1f:a5:c9:de:9f:90:4a:2f:d3:f4:69:c2.
```

@elementalvoid If you do a `molecule test` without your changes, does it successfully run testinfra? I've noticed locally that if I remove my ~/.ssh/config from the picture, doing `molecule converge; molecule verify` will fail with the error you're experiencing. However, `molecule test` will pass. I haven't looked into why, yet. I'll save that for tomorrow.

I will check that for you today. 

What I suspect is happening is that during a `molecule test`, we set environment variables for ansible-playbook, and those are also getting picked up by the ansible backend that testinfra is using. The same isn't true during a `molecule verify` which runs testinfra without calling ansible-playbook and setting up the env vars.

In particular, these env vars: https://github.com/metacloud/molecule/blob/master/molecule/ansible_playbook.py#L88-L94

which are set from these defaults:
https://github.com/metacloud/molecule/blob/master/molecule/conf/defaults.yml#L100-L105

Yes, a run using `molecule test` works perfectly fine without my change; I think you're correct about why it works. I don't use that for my normal development runs though because I'm usually adding a test and some role code then running `molecule converge && molecule idempotence && molecule verify`. I don't want to wait for `create` to run every time.

So, the question is: Should all commands which run/utilize ansible configure the environment variables? Should the environment variables be dropped in favor of setting those configurations in the ansible.cfg or ansible_inventory? Something else entirely?

Because the `ansible.cfg` is a static file and is also generated only on `create` updates to the molecule.yml won't propagate unless the vm is destroyed and recreated. This might make the environment variable more enticing, but it does mean that issues like this will continue to crop up because there are multiple places where the ansible configuration is managed.

I'm open to whatever you think is right. I have a work around for the time being.

I think another reason we haven't seen this locally is because it appears to only happen if you do a `molecule verify` >60 seconds after a `molecule converge`. That's the SSH ControlPersist timeout by default. I think most modern systems have ControlPersist enabled, but if it wasn't, this bug would be more obvious. This is what has been making it so difficult for me to reproduce consistently on my test machines.

I'm leaning toward establishing the Ansible environment variables during testinfra runs so the Ansible backend it uses is configured the same way.


Resolving patch:
diff --git a/molecule/commands.py b/molecule/commands.py
--- a/molecule/commands.py
+++ b/molecule/commands.py
@@ -377,7 +377,10 @@ def execute(self, exit=True):
             return None, None
 
         self.molecule._write_ssh_config()
-        kwargs = {'env': self.molecule._env}
+        # testinfra's Ansible calls get same env vars as ansible-playbook
+        ansible = AnsiblePlaybook(self.molecule._config.config['ansible'],
+                                  _env=self.molecule._env)
+        kwargs = {'env': ansible.env}
         kwargs['env']['PYTHONDONTWRITEBYTECODE'] = '1'
         kwargs['debug'] = True if self.molecule._args.get('--debug') else False
 


Test patch: