Repository: ansible/ansible

Issue:
hosts defined in two groups in different inventory files each is used multiple times
when using multiple inventory files defining different groups for the same hosts, ansible adds the hosts multiple times to the inventory list.

version: 1.4.4+dfsg-1 (from debian jessie)
same problem with current devel 00b3f627ebb2b8e9cf743671a9ddd2d8d8c9fe45

reproduce: 
- use files from https://gist.github.com/g-k-r/8599442
- run **ansible all --list**
- run **ansible group2 --list** and compare with **ansible type2 --list**

actual output:

``` shell
$ ansible all --list
    www.google.com
    www.yahoo.com
    www.yahoo.com
$ ansible group2 --list
    www.yahoo.com
$ ansible type2 --list
    www.yahoo.com
    www.yahoo.com
```

note when the group is defined in both files like [type1] this does not happen.

``` shell
$ ansible type1 --list
    www.google.com
```

expected output: www.yahoo.com shows up once only i.e.

``` shell
$ ansible all --list
    www.google.com
    www.yahoo.com
$ ansible group2 --list
    www.yahoo.com
$ ansible type2 --list
    www.yahoo.com
```

additional information:

to verify that tasks would actually be executed multiple times use

``` shell
$ ansible all -m debug -a '"msg=TEST {{inventory_hostname}}"'
www.google.com | success >> {
    "msg": "TEST www.google.com"
}

www.yahoo.com | success >> {
    "msg": "TEST www.yahoo.com"
}

www.yahoo.com | success >> {
    "msg": "TEST www.yahoo.com"
}
```



Discussion:
Per contributing.md can you please provide your ansible version?  (`ansible --version`)

If I remember correctly, this has been resolved in devel (1.5)

Yes, this was resolved by https://github.com/ansible/ansible/commit/559e89036b98127510794125beeaeebc82e0c141

Please rebase and try again if you are running an older version.

@sivel: amended the report

@g-k-r this is resolved in devel, and will be in the 1.5 release.

@jctanner: tested with devel, not solved, will provide unit test in PR based on devel

Reopening for testing.

@g-k-r unable to reproduce ...

```
jtanner@u1204:~/issues/5631-duplicate-task-runs$ ansible all -i inventory --list
    www.google.com
    www.yahoo.com
jtanner@u1204:~/issues/5631-duplicate-task-runs$ ansible group2 -i inventory --list
    www.yahoo.com
jtanner@u1204:~/issues/5631-duplicate-task-runs$ ansible type2 -i inventory --list
    www.yahoo.com
```

@g-k-r how do you install ansible and how are you using the checkout to verify devel's behavior?

I do the following:

``` bash
git clone https://github.com/g-k-r/ansible.git
. ./hacking/env-setup
ansible --version
make tests
```

or instead of `make tests`

``` bash
PYTHONPATH=./lib ANSIBLE_LIBRARY=./library  nosetests -d -v test/TestInventory.py:TestInventory.test_dir_inventory_multiple_groups
```

what is strange, is that testing with the original reproduction tests from the first post on this issue do work as you say, while the test started by nosetests doesn't.

```
PYTHONPATH=./lib ANSIBLE_LIBRARY=./library  nosetests -d -v test/TestInventory.py:TestInventory.test_dir_inventory_multiple_groups
test_dir_inventory_multiple_groups (TestInventory.TestInventory) ... FAIL

======================================================================
FAIL: test_dir_inventory_multiple_groups (TestInventory.TestInventory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gkr/workspaces/ansible_issue_5749/ansible/test/TestInventory.py", line 423, in test_dir_inventory_multiple_groups
    assert actual_host_names == ['zeus','morpheus']
AssertionError: 
    <ansible.inventory.group.Group object at 0x208e2c0> = <ansible.inventory.Inventory object at 0x204c510>.get_group('major-god')
    ['zeus', 'morpheus', 'zeus', 'morpheus'] = [<ansible.inventory.host.Host object at 0x208d050>.name for <ansible.inventory.host.Host object at 0x208d050> in <ansible.inventory.group.Group object at 0x208e120>.get_hosts()];
    print "%s : %s " % (<ansible.inventory.group.Group object at 0x208e120>.name, ['zeus', 'morpheus', 'zeus', 'morpheus'])
>>  assert ['zeus', 'morpheus', 'zeus', 'morpheus'] == ['zeus','morpheus'] 

-------------------- >> begin captured stdout << ---------------------
greek : ['zeus', 'morpheus', 'zeus', 'morpheus'] 

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 1 test in 0.009s
```

that is confusing, calling ansible directly succeeds:

```
cd tests
ansible greek --list -i inventory_dir
```

what doesn't work anymore is:

```
$ ansible all -i inventory_dir/ --list-hosts --limit greek
No hosts matched
```

with ansible 1.4.4 this results in:

```
$ ansible all -i inventory_dir/ --list -l greek
    zeus
    morpheus
    zeus
    morpheus
```

sorry about the many commits, i changed the test which now works identical to direct ansible calls, will now create a pull request, if you want to add the test to the code.
otherwise i think this issue is fixed already and I made a mistake during verification. 
Maybe you'd like to look into why get_group contains the hosts multiple times and why the --limit doesn't seem to work anymore, I could prepare a test again if needed.

thanks!

@g-k-r i think you are mixing up installed ansible lib paths with the checkout lib paths. Can you please retest on a clean machine with a fresh checkout and no ansible packages installed?

ok i will test in a cleanrooom VM, hopefully tonight otherwise this needs to wait until the week after next

hi @jctanner i tested in a clean debian wheezy 
only installed ansible dependencies, but not ansible itself as it is not available in pure wheezy.

to test the initial implementation with get_group i had to checkout the second commit in my pull request. since the latest commit overwrites the test implementation to use get_hosts instead of get_group. 

```
$ git clone https://github.com/g-k-r/ansible.git
...
$ git checkout --track origin/test_hosts_defined_in_multiple_groups_in_different_files
...
$ git checkout 3fe1341990babde2b3dd5295bdd3072aac88fec2
...
$ make tests
...
======================================================================
FAIL: test_dir_inventory_multiple_groups (TestInventory.TestInventory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/g-k-r/issue_5749/ansible/test/TestInventory.py", line 423, in test_dir_inventory_multiple_groups
    assert actual_host_names == ['zeus','morpheus']
AssertionError: 
    <ansible.inventory.group.Group object at 0x28d5050> = <ansible.inventory.Inventory object at 0x2736510>.get_group('major-god')
    ['zeus', 'morpheus', 'zeus', 'morpheus'] = [<ansible.inventory.host.Host object at 0x28d2c68>.name for <ansible.inventory.host.Host object at 0x28d2c68> in <ansible.inventory.group.Group object at 0x2848e88>.get_hosts()];
    print "%s : %s " % (<ansible.inventory.group.Group object at 0x2848e88>.name, ['zeus', 'morpheus', 'zeus', 'morpheus'])
>>  assert ['zeus', 'morpheus', 'zeus', 'morpheus'] == ['zeus','morpheus'] 

-------------------- >> begin captured stdout << ---------------------
greek : ['zeus', 'morpheus', 'zeus', 'morpheus'] 

--------------------- >> end captured stdout << ----------------------
```

inventory.get_group is not behaving like i though it should. anyway the get_hosts version (HEAD of the pull request) works.

It looks like there isn't any outstanding issues with this ticket so I am going to close it out based on your final comment here. I don't see the duplication that you are pointing out in the recent branches.

If you are continuing to see problems feel free to ping me @ansible.com and we can sort it out.

This sounds like this comment was about a pull request that wasn't applied yet so I'm reopening this one for now, unless we can't reproduce this on head, etc.

ok beat me to it, thanks!

Pertaining to the original description of this thread as of 1.5 this is fixed and tests correct.
https://gist.github.com/risaacson/db81aa24b61455bd32e4

After comment 11 you have a few commits that end up being a part of GH-5837. We have made some changes around I cherry picked the changes off into a branch that I am working on un-commenting test_dir_inventory. Your test should be merged as a part of GH-6287. In which you will get credit for the test code.

This issue has wondered a bit but from what I can see there is one remaining question. That question is: Should get_groups return multiple entries?

The answer to this looks to be that based on how it is used internally in inventory/**init**.py that there is a level of reduplication that ends up happening so it isn’t an issue unless you are calling it directly.

So once my PR goes in it will pull in your PR. I hope that I have shed a little bit of light on this.

@risaacson thanks, information round up appreciated


Resolving patch:
diff --git a/lib/ansible/inventory/ini.py b/lib/ansible/inventory/ini.py
--- a/lib/ansible/inventory/ini.py
+++ b/lib/ansible/inventory/ini.py
@@ -123,12 +123,22 @@ def _parse_base_groups(self):
                                 (k,v) = t.split("=", 1)
                             except ValueError, e:
                                 raise errors.AnsibleError("Invalid ini entry: %s - %s" % (t, str(e)))
-                            try:
-                                host.set_variable(k,ast.literal_eval(v))
-                            except:
-                                # most likely a string that literal_eval
-                                # doesn't like, so just set it
-                                host.set_variable(k,v)
+
+                            # If there is a hash in the value don't pass it through to ast at ast will split at the hash.
+                            if "#" in v:
+                                host.set_variable(k, v)
+                            else:
+                                try:
+                                    host.set_variable(k,ast.literal_eval(v))
+                                # Using explicit exceptions.
+                                # Likely a string that literal_eval does not like. We wil then just set it.
+                                except ValueError:
+                                    # For some reason this was thought to be malformed.
+                                    host.set_variable(k, v)
+                                except SyntaxError:
+                                    # Is this a hash with an equals at the end?
+                                    host.set_variable(k, v)
+
                     self.groups[active_group_name].add_host(host)
 
     # [southeast:children]


Test patch:
diff --git a/test/units/TestInventory.py b/test/units/TestInventory.py
--- a/test/units/TestInventory.py
+++ b/test/units/TestInventory.py
@@ -417,15 +417,24 @@ def test_allows_equals_sign_in_var(self):
         auth = inventory.get_variables('neptun')['auth']
         assert auth == 'YWRtaW46YWRtaW4='
 
-    # test disabled as needs to be updated to model desired behavior
-    #
-    #def test_dir_inventory(self):
-    #    inventory = self.dir_inventory()
-    #    vars = inventory.get_variables('zeus')
-    #
-    #   print "VARS=%s" % vars
-    #
-    #    assert vars == {'inventory_hostname': 'zeus',
-    #                    'inventory_hostname_short': 'zeus',
-    #                    'group_names': ['greek', 'major-god', 'ungrouped'],
-    #                    'var_a': '1#2'}
+    def test_dir_inventory(self):
+        inventory = self.dir_inventory()
+
+        host_vars = inventory.get_variables('zeus')
+
+        expected_vars = {'inventory_hostname': 'zeus',
+                         'inventory_hostname_short': 'zeus',
+                         'group_names': ['greek', 'major-god', 'ungrouped'],
+                         'var_a': '3#4'}
+
+        print "HOST     VARS=%s" % host_vars
+        print "EXPECTED VARS=%s" % expected_vars
+
+        assert host_vars == expected_vars
+
+    def test_dir_inventory_multiple_groups(self):
+        inventory = self.dir_inventory()
+        group_greek = inventory.get_hosts('greek')
+        actual_host_names = [host.name for host in group_greek]
+        print "greek : %s " % actual_host_names
+        assert actual_host_names == ['zeus', 'morpheus']
diff --git a/test/units/inventory_test_data/inventory_dir/0hosts b/test/units/inventory_test_data/inventory_dir/0hosts
--- a/test/units/inventory_test_data/inventory_dir/0hosts
+++ b/test/units/inventory_test_data/inventory_dir/0hosts
@@ -1,3 +1,3 @@
-zeus var_a=2
+zeus var_a=0
 morpheus
 thor
diff --git a/test/units/inventory_test_data/inventory_dir/2levels b/test/units/inventory_test_data/inventory_dir/2levels
--- a/test/units/inventory_test_data/inventory_dir/2levels
+++ b/test/units/inventory_test_data/inventory_dir/2levels
@@ -1,5 +1,5 @@
 [major-god]
-zeus var_a=1
+zeus var_a=2
 thor
 
 [minor-god]
diff --git a/test/units/inventory_test_data/inventory_dir/3comments b/test/units/inventory_test_data/inventory_dir/3comments
--- a/test/units/inventory_test_data/inventory_dir/3comments
+++ b/test/units/inventory_test_data/inventory_dir/3comments
@@ -1,5 +1,5 @@
 [major-god] # group with inline comments
-zeus var_a="1#2" # host with inline comments and "#" in the var string
+zeus var_a="3\#4" # host with inline comments and "#" in the var string
 # A comment
 thor