Repository: ansible/ansible

Issue:
Hosts from sub-groups do not keep their ordering
##### SUMMARY
A group's hosts are not returned in an ordering that is deterministic and predictable in scenarios where more than 1 group out of the group's sub-groups and itself have hosts.

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
lib/ansible/inventory/group.py

##### ANSIBLE VERSION
```
ansible --version
ansible 2.7.0.dev0
  config file = None
  configured module search path = [u'/Users/alancoding/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/alancoding/.virtualenvs/ansible/lib/python2.7/site-packages/ansible-2.7.0.dev0-py2.7.egg/ansible
  executable location = /Users/alancoding/.virtualenvs/ansible/bin/ansible
  python version = 2.7.11 (default, Oct 17 2016, 14:59:40) [GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]
```

##### CONFIGURATION
```
ansible-config dump --only-changed
ANSIBLE_NOCOWS(env: ANSIBLE_NOCOWS) = True
```

##### OS / ENVIRONMENT
Mac OS, all local actions because this has to do with the inventory loader

##### STEPS TO REPRODUCE
playbook `unordered.yml`
```yaml
# cat unordered.yml 
- hosts: localhost
  gather_facts: no
  tasks:
  - name: "top_group hosts"
    debug: var=groups['top_group']
    run_once: true
```

with inventory `hosts` in current directory

```
# cat hosts
localhost

[top_group]
test01

[g1]
host1

[g2]
host2

[g3]
host3

[g4]
host4

[top_group:children]
g1
g2
g3
g4
```

run command

```
ansible-playbook -c local -i hosts unordered.yml
```

##### EXPECTED RESULTS
the hosts which are members of the `top_group` group should come in order

```
test01
host1
host2
host3
host4
```

This is assuming that we return its own hosts first. There is another arguable option where `test01` comes last, which would actually be more consistent with how it has historically be done, but I think that giving it first is better. In my sketch, this is returning hosts by the algorithm:

 - a group's direct hosts comes before hosts of its children and grandchildren groups
 - a group's decedents are processed before moving onto the next group
 - groups are processed in ordering of the group's listed children (thus, 1, 2, 3, 4 here)
 - multiple hosts within a group are keep their original ordering

But with some of these rules, there is room for debate.

##### ACTUAL RESULTS
Right now, the ordering is not meaningful
```
# ansible-playbook -c local -i hosts unordered.yml 

PLAY [localhost] ********************************************************************************************************************************************************************

TASK [top_group hosts] **************************************************************************************************************************************************************
ok: [localhost] => {
    "groups['top_group']": [
        "host3", 
        "host1", 
        "host4", 
        "host2", 
        "test01"
    ]
}

PLAY RECAP **************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0   

```



Discussion:

Resolving patch:
diff --git a/lib/ansible/inventory/group.py b/lib/ansible/inventory/group.py
--- a/lib/ansible/inventory/group.py
+++ b/lib/ansible/inventory/group.py
@@ -82,7 +82,7 @@ def deserialize(self, data):
             g.deserialize(parent_data)
             self.parent_groups.append(g)
 
-    def _walk_relationship(self, rel):
+    def _walk_relationship(self, rel, include_self=False, preserve_ordering=False):
         '''
         Given `rel` that is an iterable property of Group,
         consitituting a directed acyclic graph among all groups,
@@ -98,21 +98,34 @@ def _walk_relationship(self, rel):
         '''
         seen = set([])
         unprocessed = set(getattr(self, rel))
+        if include_self:
+            unprocessed.add(self)
+        if preserve_ordering:
+            ordered = [self] if include_self else []
+            ordered.extend(getattr(self, rel))
 
         while unprocessed:
             seen.update(unprocessed)
-            unprocessed = set(chain.from_iterable(
-                getattr(g, rel) for g in unprocessed
-            ))
-            unprocessed.difference_update(seen)
+            new_unprocessed = set([])
 
+            for new_item in chain.from_iterable(getattr(g, rel) for g in unprocessed):
+                new_unprocessed.add(new_item)
+                if preserve_ordering:
+                    if new_item not in seen:
+                        ordered.append(new_item)
+
+            new_unprocessed.difference_update(seen)
+            unprocessed = new_unprocessed
+
+        if preserve_ordering:
+            return ordered
         return seen
 
     def get_ancestors(self):
         return self._walk_relationship('parent_groups')
 
-    def get_descendants(self):
-        return self._walk_relationship('child_groups')
+    def get_descendants(self, **kwargs):
+        return self._walk_relationship('child_groups', **kwargs)
 
     @property
     def host_names(self):
@@ -215,7 +228,7 @@ def _get_hosts(self):
 
         hosts = []
         seen = {}
-        for kid in self.get_descendants():
+        for kid in self.get_descendants(include_self=True, preserve_ordering=True):
             kid_hosts = kid.hosts
             for kk in kid_hosts:
                 if kk not in seen:
@@ -223,12 +236,6 @@ def _get_hosts(self):
                     if self.name == 'all' and kk.implicit:
                         continue
                     hosts.append(kk)
-        for mine in self.hosts:
-            if mine not in seen:
-                seen[mine] = 1
-                if self.name == 'all' and mine.implicit:
-                    continue
-                hosts.append(mine)
         return hosts
 
     def get_vars(self):


Test patch:
diff --git a/test/units/plugins/inventory/test_group.py b/test/units/plugins/inventory/test_group.py
--- a/test/units/plugins/inventory/test_group.py
+++ b/test/units/plugins/inventory/test_group.py
@@ -76,6 +76,33 @@ def test_loop_detection(self):
         with self.assertRaises(AnsibleError):
             C.add_child_group(A)
 
+    def test_direct_host_ordering(self):
+        """Hosts are returned in order they are added
+        """
+        group = Group('A')
+        # host names not added in alphabetical order
+        host_name_list = ['z', 'b', 'c', 'a', 'p', 'q']
+        expected_hosts = []
+        for host_name in host_name_list:
+            h = Host(host_name)
+            group.add_host(h)
+            expected_hosts.append(h)
+        assert group.get_hosts() == expected_hosts
+
+    def test_sub_group_host_ordering(self):
+        """With multiple nested groups, asserts that hosts are returned
+        in deterministic order
+        """
+        top_group = Group('A')
+        expected_hosts = []
+        for name in ['z', 'b', 'c', 'a', 'p', 'q']:
+            child = Group('group_{0}'.format(name))
+            top_group.add_child_group(child)
+            host = Host('host_{0}'.format(name))
+            child.add_host(host)
+            expected_hosts.append(host)
+        assert top_group.get_hosts() == expected_hosts
+
     def test_populates_descendant_hosts(self):
         A = Group('A')
         B = Group('B')