Repository: ansible/ansible

Issue:
ansible-pull is not working anymore with ansible2.4

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report


##### COMPONENT NAME
ansible-pull

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.0.0
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.5/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.5.3 (default, Jan 19 2017, 14:11:04) [GCC 6.3.0 20170118]

```

##### CONFIGURATION
```
root@stretch:/tmp# ansible-config dump --only-changed
ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR) = True
```

##### OS / ENVIRONMENT
<!---
on a debian stretch docker (image: debian:stretch) , installed via : 
```
apt-get update
apt-get -y install python3-pip git python-minimal
pip3 install ansible
```

-->

##### SUMMARY
i'm using ansible on automatic way via ansible-pull

##### STEPS TO REPRODUCE
Start on a fresh OS, ansible-pull normally download the repo (and the incentory list) via git , now it's failed as inventory file not found before lunching the ansible-pull : 
```
/usr/local/bin/ansible-pull --url=https://user:mdp@url/ansible.git --directory=/tmp/ansible/ --checkout=dev --inventory-file=/tmp/ansible/list_ami site.yml
Starting Ansible Pull at 2017-09-20 16:32:48
/usr/local/bin/ansible-pull --url=https://user:mdp@url/ansible.git --directory=/tmp/ansible/ --checkout=dev --inventory-file=/tmp/ansible/list_ami site.yml
 [WARNING]: /tmp/ansible/site.yml: File does not exist

 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: Could not match supplied host pattern, ignoring: all

 [WARNING]: provided hosts list is empty, only localhost is available

 [WARNING]: Could not match supplied host pattern, ignoring: dockerstretch

 [WARNING]: No hosts matched, nothing to do

Usage: ansible-pull -U <repository> [options] [<playbook.yml>]

pulls playbooks from a VCS repo and executes them for the local host

Options:
  --accept-host-key     adds the hostkey for the repo url if not already added
  --ask-vault-pass      ask for vault password
  --check               don't make any changes; instead, try to predict some
                        of the changes that may occur
  -C CHECKOUT, --checkout=CHECKOUT
                        branch/tag/commit to checkout. Defaults to behavior of
                        repository module.
  --clean               modified files in the working repository will be
                        discarded
  -d DEST, --directory=DEST
                        directory to checkout repository to
  -e EXTRA_VARS, --extra-vars=EXTRA_VARS
                        set additional variables as key=value or YAML/JSON, if
                        filename prepend with @
  -f, --force           run the playbook even if the repository could not be
                        updated
  --full                Do a full clone, instead of a shallow one.
  -h, --help            show this help message and exit
  -i INVENTORY, --inventory=INVENTORY, --inventory-file=INVENTORY
                        specify inventory host path
                        (default=[['/etc/ansible/hosts']]) or comma separated
                        host list. --inventory-file is deprecated
  -l SUBSET, --limit=SUBSET
                        further limit selected hosts to an additional pattern
  --list-hosts          outputs a list of matching hosts; does not execute
                        anything else
  -m MODULE_NAME, --module-name=MODULE_NAME
                        Repository module name, which ansible will use to
                        check out the repo. Default is git.
  -M MODULE_PATH, --module-path=MODULE_PATH
                        prepend colon-separated path(s) to module library
                        (default=['/root/.ansible/plugins/modules',
                        '/usr/share/ansible/plugins/modules'])
  --new-vault-id=NEW_VAULT_ID
                        the new vault identity to use for rekey
  --new-vault-password-file=NEW_VAULT_PASSWORD_FILES
                        new vault password file for rekey
  -o, --only-if-changed
                        only run the playbook if the repository has been
                        updated
  --purge               purge checkout after playbook run
  --skip-tags=SKIP_TAGS
                        only run plays and tasks whose tags do not match these
                        values
  -s SLEEP, --sleep=SLEEP
                        sleep for random interval (between 0 and n number of
                        seconds) before starting. This is a useful way to
                        disperse git requests
  -t TAGS, --tags=TAGS  only run plays and tasks tagged with these values
  --track-subs          submodules will track the latest changes. This is
                        equivalent to specifying the --remote flag to git
                        submodule update
  -U URL, --url=URL     URL of the playbook repository
  --vault-id=VAULT_IDS  the vault identity to use
  --vault-password-file=VAULT_PASSWORD_FILES
                        vault password file
  -v, --verbose         verbose mode (-vvv for more, -vvvv to enable
                        connection debugging)
  --verify-commit       verify GPG signature of checked out commit, if it
                        fails abort running the playbook. This needs the
                        corresponding VCS module to support such an operation
  --version             show program's version number and exit

  Connection Options:
    control as whom and how to connect to hosts

    -k, --ask-pass      ask for connection password
    --private-key=PRIVATE_KEY_FILE, --key-file=PRIVATE_KEY_FILE
                        use this file to authenticate the connection
    -u REMOTE_USER, --user=REMOTE_USER
                        connect as this user (default=None)
    -c CONNECTION, --connection=CONNECTION
                        connection type to use (default=smart)
    -T TIMEOUT, --timeout=TIMEOUT
                        override the connection timeout in seconds
                        (default=10)
    --ssh-common-args=SSH_COMMON_ARGS
                        specify common arguments to pass to sftp/scp/ssh (e.g.
                        ProxyCommand)
    --sftp-extra-args=SFTP_EXTRA_ARGS
                        specify extra arguments to pass to sftp only (e.g. -f,
                        -l)
    --scp-extra-args=SCP_EXTRA_ARGS
                        specify extra arguments to pass to scp only (e.g. -l)
    --ssh-extra-args=SSH_EXTRA_ARGS
                        specify extra arguments to pass to ssh only (e.g. -R)

  Privilege Escalation Options:
    control how and which user you become as on target hosts

    --ask-sudo-pass     ask for sudo password (deprecated, use become)
    --ask-su-pass       ask for su password (deprecated, use become)
    -K, --ask-become-pass
                        ask for privilege escalation password
ERROR! Could not find a playbook to run.
```

with previous version ( 2.3.2.0 ) it's was working as expected

my workarround is to download the repo with git before the ansible-pull :  
```
git clone --branch dev https://user:mdp@url/ansible.git /tmp/ansible/

```

<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
```


##### EXPECTED RESULTS
ansilbe should run as expected 

##### ACTUAL RESULTS
```
 [WARNING]: /tmp/ansible/site.yml: File does not exist
```



Discussion:
We see the same error here on Ubuntu
I believe this is likely due to the problem described in https://github.com/ansible/ansible/issues/30663

Extension matching for inventory with python3 is broken.
@sivel, no that one is that the file is found, but YAML plugin does not consider it valid (`verify_file` method), this one .. file is not found at all
@roumano the file is in the repo you are checking out? it does not exist beforehand?

that inventory would not be usable by ansible-pull, but it would be passed into ansible-plabyook.

could you update ticket with config information? `ansible-config dump --only-changed`
@boca : 
i don't have make change on the machine, it's start on a "standard" docker without change on ansible configuration, i juste make a export of one variable to have the color : 
ansible-config dump --only-changed
```
ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR) = True
```
 ( i use it to have color even if running from a .gitlab-ci.yml ) 

Exact, before running then ansible-pull, the inventory file is not exist , event the ansible working directory is not exist ( /tmp/ansible/ in my case ) 
uggg. just ran into this on all my ubuntu nodes after they auto updated themselves to ansible 2.4, and it appears to be occuring with python 2.7.

```ubuntu@dns0-sb0-ca0:~$ ansible --version
ansible 2.4.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/ubuntu/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
ubuntu@dns0-sb0-ca0:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial
ubuntu@dns0-sb0-ca0:~$ sudo ansible-pull -U git@github.com:saysayk/zork --accept-host-key -C de5f0894fff4d7e2b7bfde41c9ff0d6464d4ae26 -d /var/ansible/zork -i /var/ansible/zork/hosts_sandbox
Starting Ansible Pull at 2017-09-23 00:39:30
/usr/bin/ansible-pull -U git@github.com:saysayk/zork --accept-host-key -C de5f0894fff4d7e2b7bfde41c9ff0d6464d4ae26 -d /var/ansible/zork -i /var/ansible/zork/hosts_sandbox
 [WARNING]: Could not match supplied host pattern, ignoring: all
 [WARNING]: provided hosts list is empty, only localhost is available
 [WARNING]: Could not match supplied host pattern, ignoring: dns0-sb0-ca0
 [WARNING]: No hosts matched, nothing to do
 [WARNING]: /var/ansible/zork/dns0-sb0-ca0.yml: File does not exist /var/ansible/zork/dns0-sb0-ca0.yml: File does not exist /var/ansible/zork/local.yml: File
does not exist
```
@SudoBasher your is just a warning, seems similar but unrelated as the original issue is the error:

`ERROR! Could not find a playbook to run`
@roumano I can reproduce the issue, it is probably related to the inventory changes
@roumano I reproduced this as well. Tested in a virtualenv as well as using packages, so its not tied to Python version.

I installed Ansible 2.3.2.0 in the virtualenv and the same ansible-pull command works. For now until update, I'm locking on 2.3.2.0 (which is what I pulled down from pip).

By the way, tested both Py3 and Py2.7.12
plunked around with this more, and yes, the preceding `git clone` operation gets things started, then ansible-pull will start working correctly. after the `git clone` is done, i can change the branch sha to whatever i want, and it will download the correct github repo rev.

however, if there is no actual repo already existing on the local node where you point your `ansible-pull` target to download to (in my case `/var/ansible/zork`), then ansible-pull doesn't seem to be able to grab anything from github. one `git clone` op and it kick starts the motor.
@SudoBasher That information is good to know. Thanks for posting it. I'm not sure how feasible this will be for our deployment, but helpful that we have a workaround to the problem now.
@jjahns i'd be willing to guide you through how to manage it. our infrastructure is spanning multiple continents and we only use ansible-pull. if you need to shoot me a PM, feel free.
@SudoBasher Thanks. It sounds straightforward, so not too concerned.
same here, we're locking ansible to 2.3.2 :cry: 
In my case, ansible-pull was failing because I'm referencing an inventory file which exists _after_ the git clone operation, and this caused it to run without passing any inventory at all. This particular code path does not behave as the comment above it indicates:

https://github.com/ansible/ansible/blob/8f97aef1a365cbbbb822d6d09f96af17a076b295/lib/ansible/cli/pull.py#L156-L166
pull itself is not the one failing, it is `ansible` (called by pull) when being asked to do the git checkout:
i.e, my hostname 'erebos'
```EXEC: /home/bcoca/work/ansible/bin/ansible  -c local  -vvvvv -m git -a "name=https://github.com/bcoca/configme.git dest=/var/tmp/test depth=1" all -l "localhost,erebos,erebos.briancoca.com,127.0.0.1"   ```

which 'should work'TM .. as localhost IS available and part of the lmit .. but is not matching 'all'

Resolving patch:
diff --git a/lib/ansible/cli/pull.py b/lib/ansible/cli/pull.py
--- a/lib/ansible/cli/pull.py
+++ b/lib/ansible/cli/pull.py
@@ -67,6 +67,21 @@ class PullCLI(CLI):
                                  "look for a playbook based on the host's fully-qualified domain name,"
                                  'on the host hostname and finally a playbook named *local.yml*.', }
 
+    def _get_inv_cli(self):
+
+        inv_opts = ''
+        if getattr(self.options, 'inventory'):
+            for inv in self.options.inventory:
+                if isinstance(inv, list):
+                    inv_opts += " -i '%s' " % ','.join(inv)
+                elif ',' in inv or os.path.exists(inv):
+                    inv_opts += ' -i %s ' % inv
+
+        if not inv_opts:
+            inv_opts = " -i localhost, "
+
+        return inv_opts
+
     def parse(self):
         ''' create an options parser for bin/ansible '''
 
@@ -158,15 +173,7 @@ def run(self):
 
         # Attempt to use the inventory passed in as an argument
         # It might not yet have been downloaded so use localhost as default
-        inv_opts = ''
-        if getattr(self.options, 'inventory'):
-            for inv in self.options.inventory:
-                if isinstance(inv, list):
-                    inv_opts += " -i '%s' " % ','.join(inv)
-                elif ',' in inv or os.path.exists(inv):
-                    inv_opts += ' -i %s ' % inv
-        else:
-            inv_opts = "-i 'localhost,'"
+        inv_opts = self._get_inv_cli()
 
         # FIXME: enable more repo modules hg/svn?
         if self.options.module_name == 'git':
@@ -231,8 +238,7 @@ def run(self):
         if self.options.vault_password_files:
             for vault_password_file in self.options.vault_password_files:
                 cmd += " --vault-password-file=%s" % vault_password_file
-        if inv_opts:
-            cmd += ' %s' % inv_opts
+
         for ev in self.options.extra_vars:
             cmd += ' -e "%s"' % ev
         if self.options.ask_sudo_pass or self.options.ask_su_pass or self.options.become_ask_pass:
@@ -250,6 +256,11 @@ def run(self):
 
         os.chdir(self.options.dest)
 
+        # redo inventory options as new files might exist now
+        inv_opts = self._get_inv_cli()
+        if inv_opts:
+            cmd += inv_opts
+
         # RUN THE PLAYBOOK COMMAND
         display.debug("running ansible-playbook to do actual work")
         display.debug('EXEC: %s' % cmd)


Test patch: