Repository: ansible/ansible

Issue:
Maximum recursion depth exceeded while using include_role
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and devel branch are affected too.
Always add information AFTER of these html comments.
-->

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!---
Insert below this comment the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path
-->

include_role

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
$ ansible --version
ansible 2.5.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home_local/ansible/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
```
$ ansible-config dump --only-changed
DEFAULT_ROLES_PATH(/etc/ansible/ansible.cfg) = [u'/etc/ansible/roles', u'/usr/share/ansible/roles']
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->

Running ansible from CentOS 7.4, against itself (localhost):

```
$ cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 
```

##### SUMMARY
A limited number of ``include_role`` statements (in combination with some other includes) cause following exception:

```
RuntimeError: maximum recursion depth exceeded
```

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->

``test.yml``
```yaml
- hosts: localhost
  tasks:
      - include_tasks: "test_role1.yml"
```

``test_role1.yml``
```yaml
---
   - name: Include 1
     include_role: 
        name: "{{ test_role }}"
   
   - name: Include 2
     include_role: 
        name: "{{ test_role }}"

   - name: Include 3
     include_role: 
        name: "{{ test_role }}"

   - name: Include 4
     include_role: 
        name: "{{ test_role }}"

   - name: Include 5
     include_role: 
        name: "{{ test_role }}"

   - name: Include 6
     include_role: 
        name: "{{ test_role }}"
   
   - name: Include 7
     include_role: 
        name: "{{ test_role }}"

   - name: Include 8
     include_role: 
        name: "{{ test_role }}"

   - name: Include 9
     include_role: 
        name: "{{ test_role }}"

   - name: Include 10
     include_role: 
        name: "{{ test_role }}"

   - name: Include 11
     include_role: 
        name: "{{ test_role }}"

   - name: Include 12
     include_role: 
        name: "{{ test_role }}"

   - name: Include 13
     include_role: 
        name: "{{ test_role }}"

   - name: Include 14
     include_role: 
        name: "{{ test_role }}"

   - name: Include 15
     include_role: 
        name: "{{ test_role }}"
```

``roles/role1/tasks/main.yml``
```yaml
---
- import_tasks: "../helper/tasks.yml"
```

``helper/tasks.yml``
```yaml
- debug:
       msg: "Test"
```

``hosts``
```
[local]
localhost       ansible_connection=local
```

Directory structure with above files looks as follows:

```
|-- helper
|   `-- tasks.yml
|-- hosts
|-- roles
|   `-- role1
|       `-- tasks
|           `-- main.yml
|-- test_role1.yml
|-- test.yml
```

With above files, reproduce as follows:

```
ansible-playbook -i hosts test.yml --extra-vars "test_role=role1"
```

##### EXPECTED RESULTS
Output:

```
TASK [Include 11] **************************************************************************************************************************************************************************

TASK [role1 : debug] ***********************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Test"
}
[...]
TASK [Include 15] **************************************************************************************************************************************************************************

TASK [role1 : debug] ***********************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Test"
}
```

##### ACTUAL RESULTS
Run does not get past "Include 14".

<!--- Paste verbatim command output between quotes below -->
```
[...]
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task_include.py", line 58, in copy
    new_me = super(TaskInclude, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 351, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task_include.py", line 58, in copy
    new_me = super(TaskInclude, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 351, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/role_include.py", line 135, in copy
    new_me = super(IncludeRole, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task_include.py", line 58, in copy
    new_me = super(TaskInclude, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 351, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task_include.py", line 58, in copy
    new_me = super(TaskInclude, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 351, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/role_include.py", line 135, in copy
    new_me = super(IncludeRole, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task_include.py", line 58, in copy
    new_me = super(TaskInclude, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 351, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task_include.py", line 58, in copy
    new_me = super(TaskInclude, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 351, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/role_include.py", line 135, in copy
    new_me = super(IncludeRole, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task_include.py", line 58, in copy
    new_me = super(TaskInclude, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 351, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task_include.py", line 58, in copy
    new_me = super(TaskInclude, self).copy(exclude_parent=exclude_parent, exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 351, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 206, in copy
    new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 196, in copy
    new_me = super(Block, self).copy()
  File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 340, in copy
    new_me = self.__class__()
  File "/usr/lib/python2.7/site-packages/ansible/playbook/block.py", line 63, in __init__
    super(Block, self).__init__()
  File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 198, in __init__
    self.vars = dict()
RuntimeError: maximum recursion depth exceeded
```



Discussion:
Files identified in the description:
* [lib/ansible/playbook/role/include.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/playbook/role/include.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
I can confirm this happens on py2, but does not affect py3.

We have done a lot to improve recursion limits, we'll need to dig into this further.
Thank you for the very quick triage!

I should also point out that the example is a simplification of real code where the problem occurs with only 5 role includes. It is probably possible to hit the problem with even less includes, if the surrounding code is made sufficiently complex/large.

Any tips to work around this? Should we consider moving to py3? Should we avoid the use of ``include_role``? Perhaps the limits of ``include_role`` are already documented somewhere?
Depending on how things are structured, there are different limits to `include_role`.  With recent improvements a recursive `include_role` has been shown to work up to a depth of roughly 200.

I don't have any stats related to using `include_tasks` along with `include_role`, which may have a different trigger.

We do not have any documentation of `include_role` limitations at this time, but it is known that this problem can occurs under different situations. We have been working to reduce those limitations.

But from my tests python3 does seem to improve performance and recursion depth issues.
With python3, I manage to get to a depth of 16 with the above test code.

Interestingly, if I replace the ``import_tasks`` in ``roles/role1/tasks/main.yml`` by ``include_tasks``, I get to a depth of 40. Doing this with python3 allows going beyond 90 (did not test up to the limit).

Replacing ``import_tasks`` by ``include_tasks`` also fixes the issue in our real code for now.


If I divide the tasks in role1 into 2 files such as role2 and role3, each having half the include_tasks from role1 and then include or import them instead of role1 it works - this is not ideal but is the workaround I am using.
cc @sivel @bcoca @jimi-c @samdoran 
Just a note that this could very well be related to the reparenting work done in `ansible.playbook.block.Block.copy->_dupe_task_list`

Issues were resolved in there before, to make sure we re-parented properly. But the testing done there was just with all the same include type.  Unsure how mixing different include types may affect things there.
I feel pretty confident that this is a `_dupe_task_list` issue.  Oddly, I am seeing a strange difference between py2 and py3 in comparing blocks.
I've found the source of the py2 vs py3 difference.  https://github.com/ansible/ansible/pull/38687 has been submitted to resolve that.

As noted earlier, it allows you to get minimally farther, before getting a recursion error.  We'll have additional work to do, to see what can be done with that.

Resolving patch:
diff --git a/lib/ansible/executor/play_iterator.py b/lib/ansible/executor/play_iterator.py
--- a/lib/ansible/executor/play_iterator.py
+++ b/lib/ansible/executor/play_iterator.py
@@ -524,7 +524,7 @@ def _insert_tasks_into_state(self, state, task_list):
             if state.tasks_child_state:
                 state.tasks_child_state = self._insert_tasks_into_state(state.tasks_child_state, task_list)
             else:
-                target_block = state._blocks[state.cur_block].copy(exclude_parent=True)
+                target_block = state._blocks[state.cur_block].copy()
                 before = target_block.block[:state.cur_regular_task]
                 after = target_block.block[state.cur_regular_task:]
                 target_block.block = before + task_list + after
@@ -533,7 +533,7 @@ def _insert_tasks_into_state(self, state, task_list):
             if state.rescue_child_state:
                 state.rescue_child_state = self._insert_tasks_into_state(state.rescue_child_state, task_list)
             else:
-                target_block = state._blocks[state.cur_block].copy(exclude_parent=True)
+                target_block = state._blocks[state.cur_block].copy()
                 before = target_block.rescue[:state.cur_rescue_task]
                 after = target_block.rescue[state.cur_rescue_task:]
                 target_block.rescue = before + task_list + after
@@ -542,7 +542,7 @@ def _insert_tasks_into_state(self, state, task_list):
             if state.always_child_state:
                 state.always_child_state = self._insert_tasks_into_state(state.always_child_state, task_list)
             else:
-                target_block = state._blocks[state.cur_block].copy(exclude_parent=True)
+                target_block = state._blocks[state.cur_block].copy()
                 before = target_block.always[:state.cur_always_task]
                 after = target_block.always[state.cur_always_task:]
                 target_block.always = before + task_list + after
diff --git a/lib/ansible/playbook/block.py b/lib/ansible/playbook/block.py
--- a/lib/ansible/playbook/block.py
+++ b/lib/ansible/playbook/block.py
@@ -69,6 +69,10 @@ def __eq__(self, other):
         '''object comparison based on _uuid'''
         return self._uuid == other._uuid
 
+    def __ne__(self, other):
+        '''object comparison based on _uuid'''
+        return self._uuid != other._uuid
+
     def get_vars(self):
         '''
         Blocks do not store variables directly, however they may be a member
@@ -173,21 +177,16 @@ def _dupe_task_list(task_list, new_block):
                 new_task = task.copy(exclude_parent=True)
                 if task._parent:
                     new_task._parent = task._parent.copy(exclude_tasks=True)
-                    # go up the parentage tree until we find an
-                    # object without a parent and make this new
-                    # block their parent
-                    cur_obj = new_task
-                    while cur_obj._parent:
-                        if cur_obj._parent:
-                            prev_obj = cur_obj
-                        cur_obj = cur_obj._parent
-
-                    # Ensure that we don't make the new_block the parent of itself
-                    if cur_obj != new_block:
-                        cur_obj._parent = new_block
+                    if task._parent == new_block:
+                        # If task._parent is the same as new_block, just replace it
+                        new_task._parent = new_block
                     else:
-                        # prev_obj._parent is cur_obj, to allow for mutability we need to use prev_obj
-                        prev_obj._parent = new_block
+                        # task may not be a direct child of new_block, search for the correct place to insert new_block
+                        cur_obj = new_task._parent
+                        while cur_obj._parent and cur_obj._parent != new_block:
+                            cur_obj = cur_obj._parent
+
+                        cur_obj._parent = new_block
                 else:
                     new_task._parent = new_block
                 new_task_list.append(new_task)
@@ -203,7 +202,7 @@ def _dupe_task_list(task_list, new_block):
 
         new_me._parent = None
         if self._parent and not exclude_parent:
-            new_me._parent = self._parent.copy(exclude_tasks=exclude_tasks)
+            new_me._parent = self._parent.copy(exclude_tasks=True)
 
         if not exclude_tasks:
             new_me.block = _dupe_task_list(self.block or [], new_me)
diff --git a/lib/ansible/playbook/role/__init__.py b/lib/ansible/playbook/role/__init__.py
--- a/lib/ansible/playbook/role/__init__.py
+++ b/lib/ansible/playbook/role/__init__.py
@@ -422,9 +422,7 @@ def compile(self, play, dep_chain=None):
             block_list.extend(dep_blocks)
 
         for idx, task_block in enumerate(self._task_blocks):
-            new_task_block = task_block.copy(exclude_parent=True)
-            if task_block._parent:
-                new_task_block._parent = task_block._parent.copy()
+            new_task_block = task_block.copy()
             new_task_block._dep_chain = new_dep_chain
             new_task_block._play = play
             if idx == len(self._task_blocks) - 1:


Test patch: