Issue 1:
ISSUE TYPE

Bug Report

COMPONENT NAME

Included / imported plays and group_vars loading

ANSIBLE VERSION

$ PYTHONPATH=~/ansible-2.4.0.0/lib ~/ansible-2.4.0.0/bin/ansible --version
ansible 2.4.0.0
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/home/centos/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /home/centos/ansible-2.4.0.0/lib/ansible
executable location = /home/centos/ansible-2.4.0.0/bin/ansible
python version = 2.7.5 (default, Nov 20 2015, 02:00:19) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
$

This is Ansible 2.4.0 RC2.

CONFIGURATION

Nothing altered.

OS / ENVIRONMENT

Centos 7.2

SUMMARY

The behavior of group_vars loading for included plays has changed between 2.3 and 2.4. Previously the group_vars would load based on the initial playbook directory; in 2.4 the group_vars loads from the included playbook's directory. The behavior difference is a breaking change, leading to failures with existing playbooks structured in particular ways.

I recognize that this breaking change may have been intended at 2.4. The documentation (http://docs.ansible.com/ansible/latest/intro_inventory.html#splitting-out-host-and-group-specific-data) refers only to "the playbook directory", which is unclear about whether that's the entry playbook or the included playbook.

STEPS TO REPRODUCE

entry.yml

---
- hosts: all
tasks:
- name: From entry.yml
debug:
var: INSTALL_USER

- include: internal/testing.yml

internal/testing.yml

---
- hosts: all
tasks:
- name: From internal/testing.yml
debug:
var: INSTALL_USER

group_vars/all

---
INSTALL_USER: testing

inventories/inventory.ini

host ansible_connection=local

Execution:

ansible-playbook -i inventories/inventory.ini entry.yml

EXPECTED RESULTS

I expected the same behavior I see in 2.3, where the group_vars/all entries are present for internal.testing.yml:

$ ansible --version
ansible 2.3.1.0
config file = /etc/ansible/ansible.cfg
configured module search path = Default w/o overrides
python version = 2.7.5 (default, Nov 20 2015, 02:00:19) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]

$ ansible-playbook -i inventories/inventory.ini entry.yml

PLAY [all] *************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************
ok: [host]

TASK [From entry.yml] **************************************************************************************************************************
ok: [host] => {
"INSTALL_USER": "testing"
}

PLAY [all] *************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************
ok: [host]

TASK [From internal/testing.yml] ***************************************************************************************************************
ok: [host] => {
"INSTALL_USER": "testing"
}

PLAY RECAP *************************************************************************************************************************************
host : ok=4 changed=0 unreachable=0 failed=0

$

ACTUAL RESULTS

$ PYTHONPATH=~/ansible-2.4.0.0/lib ~/ansible-2.4.0.0/bin/ansible --version
ansible 2.4.0.0
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/home/centos/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /home/centos/ansible-2.4.0.0/lib/ansible
executable location = /home/centos/ansible-2.4.0.0/bin/ansible
python version = 2.7.5 (default, Nov 20 2015, 02:00:19) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]

$ PYTHONPATH=~/ansible-2.4.0.0/lib ~/ansible-2.4.0.0/bin/ansible-playbook -i inventories/inventory.ini entry.yml
[DEPRECATION WARNING]: 'include' for playbook includes. You should use 'import_playbook' instead. This feature will be removed in version 2.8.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [all] *************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************
ok: [host]

TASK [From entry.yml] **************************************************************************************************************************
ok: [host] => {
"INSTALL_USER": "testing"
}

PLAY [all] *************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************
ok: [host]

TASK [From internal/testing.yml] ***************************************************************************************************************
ok: [host] => {
"INSTALL_USER": "VARIABLE IS NOT DEFINED!"
}

PLAY RECAP *************************************************************************************************************************************
host : ok=4 changed=0 unreachable=0 failed=0

$

If I create internal/testing/group_vars/all with a different INSTALL_USER value, then 2.4 uses that value while 2.3 does not:

$ mkdir internal/group_vars

$ echo "INSTALL_USER: different" > internal/group_vars/all

$ PYTHONPATH=~/ansible-2.4.0.0/lib ~/ansible-2.4.0.0/bin/ansible-playbook -i inventories/inventory.ini entry.yml
[DEPRECATION WARNING]: 'include' for playbook includes. You should use 'import_playbook' instead. This feature will be removed in version 2.8.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [all] *************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************
ok: [host]

TASK [From entry.yml] **************************************************************************************************************************
ok: [host] => {
"INSTALL_USER": "testing"
}

PLAY [all] *************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************
ok: [host]

TASK [From internal/testing.yml] ***************************************************************************************************************
ok: [host] => {
"INSTALL_USER": "different"
}

PLAY RECAP *************************************************************************************************************************************
host : ok=4 changed=0 unreachable=0 failed=0

$ ansible-playbook -i inventories/inventory.ini entry.yml

PLAY [all] *************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************
ok: [host]

TASK [From entry.yml] **************************************************************************************************************************
ok: [host] => {
"INSTALL_USER": "testing"
}

PLAY [all] *************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************
ok: [host]

TASK [From internal/testing.yml] ***************************************************************************************************************
ok: [host] => {
"INSTALL_USER": "testing"
}

PLAY RECAP *************************************************************************************************************************************
host : ok=4 changed=0 unreachable=0 failed=0

$

Issue 2:
ISSUE TYPE

Bug Report

COMPONENT NAME

Core / import_playbook

ANSIBLE VERSION

ansible 2.4.0
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/home/renji/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]

CONFIGURATION

[defaults]
log_path = ./logs/ansible.log
roles_path = /etc/ansible/roles/:roles/
jinja2_extensions = jinja2.ext.do,jinja2.ext.i18n

host_key_checking = no
retry_files_enabled = no

OS / ENVIRONMENT

Distributor ID:	Ubuntu

Description:	Ubuntu 16.04.3 LTS

Release:	16.04

Codename:	xenial

SUMMARY

Vars scope is broken for a playbook imported with import_playbook and when this playbook is located in a sub directory.

STEPS TO REPRODUCE

---
- name: Test VARS
hosts: all
connection: local
gather_facts: no
tasks:
- name: debug 1
debug: var=backup

- import_playbook: test/pb__test_import.yml

---
- name: Test VARS
hosts: all
connection: local
gather_facts: no
tasks:
- name: debug 2
debug: var=backup

EXPECTED RESULTS

ansible-playbook 2.4.0
config file = /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/ansible.cfg
configured module search path = [u'/home/renji/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
executable location = /usr/local/bin/ansible-playbook
python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
Using /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/ansible.cfg as config file
Vault password:
setting up inventory plugins
Set default localhost to localhost
Parsed /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/inventories/int5 inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: pb__test.yml ****************************************************************************************************************************************************************************************************************
2 plays in pb__test.yml

PLAY [Test VARS] **********************************************************************************************************************************************************************************************************************
META: ran handlers
Found a vault_id (default) in the vaulttext
We have a secret associated with vault id (default), will try to use to decrypt /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/group_vars/all.yaml
Trying to use vault secret=(<ansible.parsing.vault.PromptVaultSecret object at 0x7fadaafbc290>) id=default to decrypt /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/group_vars/all.yaml
Trying secret <ansible.parsing.vault.PromptVaultSecret object at 0x7fadaafbc290> for vault_id=default
decrypt succesful with secret=<ansible.parsing.vault.PromptVaultSecret object at 0x7fadaafbc290> and vault_id=default
Found a vault_id (default) in the vaulttext
We have a secret associated with vault id (default), will try to use to decrypt /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/group_vars/int5.yaml
Trying to use vault secret=(<ansible.parsing.vault.PromptVaultSecret object at 0x7fadaafbc290>) id=default to decrypt /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/group_vars/int5.yaml
Trying secret <ansible.parsing.vault.PromptVaultSecret object at 0x7fadaafbc290> for vault_id=default
decrypt succesful with secret=<ansible.parsing.vault.PromptVaultSecret object at 0x7fadaafbc290> and vault_id=default

TASK [debug 1] ************************************************************************************************************************************************************************************************************************
task path: /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/pb__test_vars.yml:7
ok: [asa-aub-i-fwloob-0000] => {
"backup": {
"diff": {
"parts": "tmp/gitDiff/",
"report": "tmp/git.diff"
},
"git": {
"repository": "https://git.corp.cloudwatt.com/noc/backup-int5.git"
},
"location": "../backups/int5",
"log": {
"down": "logs/down.log"
}
}
}
META: ran handlers
META: ran handlers

PLAY [Test VARS] **********************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [debug 2] ************************************************************************************************************************************************************************************************************************
task path: /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/pb__test_import.yml:7
ok: [asa-aub-i-fwloob-0000] => {
"backup": {
"diff": {
"parts": "tmp/gitDiff/",
"report": "tmp/git.diff"
},
"git": {
"repository": "https://git.corp.cloudwatt.com/noc/backup-int5.git"
},
"location": "../backups/int5",
"log": {
"down": "logs/down.log"
}
}
}
META: ran handlers
META: ran handlers

PLAY RECAP ****************************************************************************************************************************************************************************************************************************
asa-aub-i-fwloob-0000 : ok=2 changed=0 unreachable=0 failed=0

ACTUAL RESULTS

ansible-playbook 2.4.0
config file = /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/ansible.cfg
configured module search path = [u'/home/renji/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
executable location = /usr/local/bin/ansible-playbook
python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
Using /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/ansible.cfg as config file
Vault password:
setting up inventory plugins
Set default localhost to localhost
Parsed /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/inventories/int5 inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: pb__test.yml ****************************************************************************************************************************************************************************************************************
2 plays in pb__test.yml

PLAY [Test VARS] **********************************************************************************************************************************************************************************************************************
META: ran handlers
Found a vault_id (default) in the vaulttext
We have a secret associated with vault id (default), will try to use to decrypt /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/group_vars/all.yaml
Trying to use vault secret=(<ansible.parsing.vault.PromptVaultSecret object at 0x7f3d160de290>) id=default to decrypt /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/group_vars/all.yaml
Trying secret <ansible.parsing.vault.PromptVaultSecret object at 0x7f3d160de290> for vault_id=default
decrypt succesful with secret=<ansible.parsing.vault.PromptVaultSecret object at 0x7f3d160de290> and vault_id=default
Found a vault_id (default) in the vaulttext
We have a secret associated with vault id (default), will try to use to decrypt /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/group_vars/int5.yaml
Trying to use vault secret=(<ansible.parsing.vault.PromptVaultSecret object at 0x7f3d160de290>) id=default to decrypt /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/group_vars/int5.yaml
Trying secret <ansible.parsing.vault.PromptVaultSecret object at 0x7f3d160de290> for vault_id=default
decrypt succesful with secret=<ansible.parsing.vault.PromptVaultSecret object at 0x7f3d160de290> and vault_id=default

TASK [debug 1] ************************************************************************************************************************************************************************************************************************
task path: /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/pb__test_vars.yml:7
ok: [asa-aub-i-fwloob-0000] => {
"backup": {
"diff": {
"parts": "tmp/gitDiff/",
"report": "tmp/git.diff"
},
"git": {
"repository": "https://git.corp.cloudwatt.com/noc/backup-int5.git"
},
"location": "../backups/int5",
"log": {
"down": "logs/down.log"
}
}
}
META: ran handlers
META: ran handlers

PLAY [Test VARS] **********************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [debug 2] ************************************************************************************************************************************************************************************************************************
task path: /home/renji/scripts/cloudwatt/network/ansible/ansible-backup/test/pb__test_import.yml:7
ok: [asa-aub-i-fwloob-0000] => {
"backup": "VARIABLE IS NOT DEFINED!"
}
META: ran handlers
META: ran handlers

PLAY RECAP ****************************************************************************************************************************************************************************************************************************
asa-aub-i-fwloob-0000 : ok=2 changed=0 unreachable=0 failed=0