InventoryDir: fixing yet another corner case

I eventually bumped in yet another corner case that causes this traceback.

```
Traceback (most recent call last):
  File "/home/serge/sbin/ansible-inventory", line 190, in <module>
    cli.run()
  File "/home/serge/sbin/ansible-inventory", line 182, in run
    cli.inv_init()
  File "/home/serge/sbin/ansible-inventory", line 101, in inv_init
    self.inventory = inventory.Inventory(self.options.inventory)
  File "/usr/lib/pymodules/python2.7/ansible/inventory/__init__.py", line 100, in __init__
    self.parser = InventoryDirectory(filename=host_list)
  File "/usr/lib/pymodules/python2.7/ansible/inventory/dir.py", line 54, in __init__
    parser = InventoryDirectory(filename=fullpath)
  File "/usr/lib/pymodules/python2.7/ansible/inventory/dir.py", line 119, in __init__
    group.parent_groups.remove(allgroup)
ValueError: list.remove(x): x not in list
```