Repository: ansible/ansible

Issue:
get_mount_facts timeout is very small and not ignored as expected
I've analysed #10746 more deeply and found that my first suspition was wrong and timeout error was not really caused by lack of lsblk. Moreover, it was not ingored as intened.

The reason is that UUID getting added by #10055 broke timeout handling because of Python's subprocess module code. In _execute_child method there's an except code catching all exceptions: that's why TimeoutError is not caught in  Facts.populate and ignored.

```
failed: [SOME_HOST] => {"cmd": "/bin/lsblk -ln --output UUID /dev/sdi2", "failed": true, "rc": 257}
msg: Traceback (most recent call last):
  File "<stdin>", line 1650, in run_command
  File "/usr/lib64/python2.4/subprocess.py", line 550, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.4/subprocess.py", line 988, in _execute_child
    data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
  File "<stdin>", line 1796, in _handle_timeout
TimeoutError: Timer expired
```

Suggested solution: get UUID's without calling lsblk with subprocess.

Margin note: 10s timeout was enough before, but with lsblk get_mount_facts on my system with no network filesystems mounted lasted even 15 seconds.



Discussion:
I disagree that 10s is too little for listing UUIDs and in the many systems i've tested lsblk takes less than 1s. Keeping this open to investigate the issue about the timeouts not being handled properly.

I ran into the same issue some days ago. In my particular case I have a install-once-and-throw-away test environment where I'm using docker containers for the underlying system.

It turns out that in this case the problem was already known (see https://github.com/docker/docker/issues/12192) and that upgrading the kernel of my docker host system resolved the issue.

In our production environment, we see this come up fairly often since the servers are pretty busy and not particularly high-performance. When RAID is rebuilding it's especially common. Even if the default stays the same, it would be nice if we could override or otherwise ignore when it isn't pertinent.

We are seeing this happen a lot on CEPH nodes with LSI SAS controllers. It's extremely random.

`failed: [obj-005] => {"cmd": "/bin/lsblk -ln --output UUID /dev/sdb1", "failed": true, "rc": 257}`
`msg: Traceback (most recent call last):`
`File "<stdin>", line 1671, in run_command`
`File "<stdin>", line 1801, in _handle_timeout`
`TimeoutError: Timer expired`

When running it again the timeout may or may not occur.

@bcoca although ideally blkid should return immediately, there are cases where the timeout may need to be increased due to high system IO load (CEPH rebuilds come to mind). Please at least consider making it a variable so that it can be adjusted in ansible.cfg.

Yes, I've been thinking of adding at timeout option to setup, defaulting to current 

Any suggested workarounds to this for now?  We hit this very often on swift nodes.

@vinsh nothing other than patching ansible; for 1.9.1-1 @ b47d1d7e697bbc2346d128d7d1857e53b80a9eb9 (increase from 20 to whatever time you need): 

``` diff
--- a/lib/ansible/module_utils/facts.py
+++ b/lib/ansible/module_utils/facts.py
@@ -53,7 +53,7 @@ except ImportError:
 class TimeoutError(Exception):
     pass

-def timeout(seconds=10, error_message="Timer expired"):
+def timeout(seconds=20, error_message="Timer expired"):
     def decorator(func):
         def _handle_timeout(signum, frame):
             raise TimeoutError(error_message)
```

Thank you for the clarification, we might branch/patch in house with your suggestion for now.

Update: the reason we had this timeout was that "nofile" in limits.conf is set to vert large value on our servers (1000000) and python subprocess module was busy closing all those filedescriptors on every command. We introduced a workaround: ansible_python_inerpretter points to a wrapper script calling "ulimit -n 1024" before python. Regardless of timeout case, ignoring still doesn't work  because of exception handling in subprocess module.

Was there a conclusion on this issue? I have observed the same issue.
- Is the answer that on many systems, 10 seconds is indeed too tight of a timeout and so it should simply be increased in master?
- is the answer that many systems are improperly configured and should drop the nofile limit?

For context, I am using [Elasticluster](https://github.com/gc3-uzh-ch/elasticluster), which installs Ansible 1.9.1 as a dependency:

```
elasticluster/lib/python2.7/site-packages/ansible-1.9.1-py2.7.egg
```

Is there a workaround here, or does Elasticluster need to provide a custom build of Ansible?

I'd just like to add that busy hosts with hundreds of LVM logical volumes is also a scenario where I'm seeing intermittent failures caused by the ten second timeout. I'm now running a fork with a larger value. I can see that, because it's a decorator, making this timeout configurable may be a pain, but it would be really useful for certain use cases (while I can also see that for some people 10 seconds is a really long time, so I'm not in favour of just making it bigger).

This has also bitten me. A 10s timeout is generous on dedicated hardware, but when we provision hundreds of VMs in shared infrastructure that are busy doing disk-intensive operations, 10s is not nearly long enough. This very much should be configurable.

@thenewwazoo it is configurable. "timeout = <seconds>" in ansible.cfg

@oleyka that would definitely alter the timeout for the ssh connection, but this issue is related to the timeout decorator inside facts.py which has no hooks into any configuration. 

For folks with storage configs that are causing timeouts:
- are the ansible_mounts facts being used? 
  If so, how. In particular, is the device uuid used?
  
  My first thought is 'stop running lsblk just to add the device uuids', so looking for evidence that is a used feature.
- are you using other tools that handle this better?  
  
  Everything seems to eventually end up reading /sys/block/\* so I'm curious if there is a better tools for this. Ideally something that actually completes quickly versus something that just doesn't timeout, but either way ;->
- would filtering the types of block devices and mount points that is collected be useful? Any easy targets for skipping by default?
- Any ideas on which parts of storage info collection are troublesome?  findmnt? lsblk? All of the above?
- Anyone have example reports from troublesome systems they could share? 
  
  For example: 
  - a 'ansible -m setup' output (or interesting subsets).
    - lsblk output ( 'lsblk -ln --output UUID')
    - ls -laR /sys/block/
  
  (any variation of those or other reports is useful) 

@alikins I think, for me, part of the problem is that this fact is O(n) -- it forks a command for every entry in `/etc/mtab`, but has a constant timeout:

https://github.com/ansible/ansible/blob/4b5203c5fb00ede6790bf48fac1d926e9b626bf4/lib/ansible/module_utils/facts.py#L1216

On one of my troublesome machines, I have 96 entries in `/etc/mtab`, with 89 LVM volumes and moderately high load. Forking around 90 processes that hit the disks, one after another, on a loaded machine regularly exceeds 10 seconds. I realise my case is probably verging on pathological, so I don't expect the defaults to change, but being able to tune this would be great.

(Also I personally don't even use these facts for anything, but I am using other facts, so I can't just run with `gather_facts: False`)

My current workaround is to execute my playbook several times until the gathering of facts succeeds.

Hi!

A change has been applied for this ticket, which should resolve this item for you.

https://github.com/ansible/ansible-modules-core/pull/4093
https://github.com/ansible/ansible/pull/16551

If you believe this ticket is not resolved, or have further questions, please let us know by stopping by one of the two mailing lists, as appropriate:
- https://groups.google.com/forum/#!forum/ansible-project - for user questions, tips, and tricks
- https://groups.google.com/forum/#!forum/ansible-devel - for strategy, future planning, and questions about writing code

Because this project is very active, we're unlikely to see comments on closed tickets, though the mailing list is a great way to get involved or discuss this one further.

Thanks!

@alikins: To answer your questions on behalf of Backblaze...

Our storage pods have 45 or 60 drives, each mounted as a separate ext4 file system.  We're seeing frequent timeouts on `lsblk` with the 10 second timeout built into 2.1.   The new `gather_timeout` setting would fix the error for us, but would mean that we have to wait for `lsblk` on all the drives, which is annoying.

We would prefer just to disable gathering the uuids of the drives so things run faster. 

Based on https://github.com/ansible/ansible/issues/10779#issuecomment-238731228 I'm working on a patch that should improve this.

The gist of the patching is replacing the use of a 'lsblk' call for every mtab entry, to just making one lsblk call to collect all of the info and mapping it to the mtab info. Hopefully that will be faster (unless the single lsblk entry takes just as long...)


Resolving patch:
diff --git a/lib/ansible/module_utils/facts.py b/lib/ansible/module_utils/facts.py
--- a/lib/ansible/module_utils/facts.py
+++ b/lib/ansible/module_utils/facts.py
@@ -38,7 +38,9 @@
 except ImportError:
     # python3
     import configparser
+
 from ansible.module_utils.basic import get_all_subclasses
+from ansible.module_utils.six import PY3
 
 # py2 vs py3; replace with six via ansiballz
 try:
@@ -971,7 +973,10 @@ def __new__(cls, *arguments, **keyword):
         for sc in get_all_subclasses(Hardware):
             if sc.platform == platform.system():
                 subclass = sc
-        return super(cls, subclass).__new__(subclass, *arguments, **keyword)
+        if PY3:
+            return super(cls, subclass).__new__(subclass)
+        else:
+            return super(cls, subclass).__new__(subclass, *arguments, **keyword)
 
     def populate(self):
         return self.facts
@@ -997,6 +1002,12 @@ class LinuxHardware(Hardware):
     # Now we have all of these in a dict structure
     MEMORY_FACTS = ORIGINAL_MEMORY_FACTS.union(('Buffers', 'Cached', 'SwapCached'))
 
+    # regex used against findmnt output to detect bind mounts
+    BIND_MOUNT_RE = re.compile(r'.*\]')
+
+    # regex used against mtab content to find entries that are bind mounts
+    MTAB_BIND_MOUNT_RE = re.compile(r'.*bind.*"')
+
     def populate(self):
         self.get_cpu_facts()
         self.get_memory_facts()
@@ -1208,58 +1219,118 @@ def get_dmi_facts(self):
                 else:
                     self.facts[k] = 'NA'
 
+    def _run_lsblk(self, lsblk_path):
+        args = ['--list', '--noheadings', '--paths',  '--output', 'NAME,UUID']
+        cmd = [lsblk_path] + args
+        rc, out, err = self.module.run_command(cmd)
+        return rc, out, err
+
+    def _lsblk_uuid(self):
+        uuids = {}
+        lsblk_path = self.module.get_bin_path("lsblk")
+        if not lsblk_path:
+            return uuids
+
+        rc, out, err = self._run_lsblk(lsblk_path)
+        if rc != 0:
+            return uuids
+
+        # each line will be in format:
+        # <devicename><some whitespace><uuid>
+        # /dev/sda1  32caaec3-ef40-4691-a3b6-438c3f9bc1c0
+        for lsblk_line in out.splitlines():
+            if not lsblk_line:
+                continue
+
+            line = lsblk_line.strip()
+            fields = line.rsplit(None, 1)
+
+            if len(fields) < 2:
+                continue
+
+            device_name, uuid = fields[0].strip(), fields[1].strip()
+            if device_name in uuids:
+                continue
+            uuids[device_name] = uuid
+
+        return uuids
+
+    def _run_findmnt(self, findmnt_path):
+        args = ['--list', '--noheadings', '--notruncate']
+        cmd = [findmnt_path] + args
+        rc, out, err = self.module.run_command(cmd)
+        return rc, out, err
+
+    def _find_bind_mounts(self):
+        bind_mounts = set()
+        findmnt_path = self.module.get_bin_path("findmnt")
+        if not findmnt_path:
+            return bind_mounts
+
+        rc, out, err = self._run_findmnt(findmnt_path)
+        if rc != 0:
+            return bind_mounts
+
+        # find bind mounts, in case /etc/mtab is a symlink to /proc/mounts
+        for line in out.splitlines():
+            fields = line.split()
+            # fields[0] is the TARGET, fields[1] is the SOURCE
+            if len(fields) < 2:
+                continue
+
+            # bind mounts will have a [/directory_name] in the SOURCE column
+            if self.BIND_MOUNT_RE.match(fields[1]):
+                bind_mounts.add(fields[0])
+
+        return bind_mounts
+
+    def _mtab_entries(self):
+        mtab = get_file_content('/etc/mtab', '')
+        mtab_entries = []
+        for line in mtab.splitlines():
+            fields = line.split()
+            if len(fields) < 4:
+                continue
+            mtab_entries.append(fields)
+        return mtab_entries
+
     @timeout(10)
     def get_mount_facts(self):
-        uuids = dict()
         self.facts['mounts'] = []
-        bind_mounts = []
-        findmntPath = self.module.get_bin_path("findmnt")
-        if findmntPath:
-            rc, out, err = self.module.run_command("%s -lnur" % ( findmntPath ), use_unsafe_shell=True)
-            if rc == 0:
-                # find bind mounts, in case /etc/mtab is a symlink to /proc/mounts
-                for line in out.split('\n'):
-                    fields = line.rstrip('\n').split()
-                    if(len(fields) < 2):
-                        continue
-                    if(re.match(".*\]",fields[1])):
-                        bind_mounts.append(fields[0])
 
-        mtab = get_file_content('/etc/mtab', '')
-        for line in mtab.split('\n'):
-            fields = line.rstrip('\n').split()
-            if len(fields) < 4:
+        bind_mounts = self._find_bind_mounts()
+        uuids = self._lsblk_uuid()
+        mtab_entries = self._mtab_entries()
+
+        mounts = []
+        for fields in mtab_entries:
+            device, mount, fstype, options = fields[0], fields[1], fields[2], fields[3]
+
+            if not device.startswith('/') and ':/' not in device:
                 continue
-            if fields[0].startswith('/') or ':/' in fields[0]:
-                if(fields[2] != 'none'):
-                    size_total, size_available = self._get_mount_size_facts(fields[1])
-                    if fields[0] in uuids:
-                        uuid = uuids[fields[0]]
-                    else:
-                        uuid = 'NA'
-                        lsblkPath = self.module.get_bin_path("lsblk")
-                        if lsblkPath:
-                            rc, out, err = self.module.run_command("%s -ln --output UUID %s" % (lsblkPath, fields[0]), use_unsafe_shell=True)
-
-                            if rc == 0:
-                                uuid = out.strip()
-                                uuids[fields[0]] = uuid
-
-                    if fields[1] in bind_mounts:
-                        # only add if not already there, we might have a plain /etc/mtab
-                        if not re.match(".*bind.*", fields[3]):
-                            fields[3] += ",bind"
-
-                    self.facts['mounts'].append(
-                        {'mount': fields[1],
-                         'device':fields[0],
-                         'fstype': fields[2],
-                         'options': fields[3],
-                         # statvfs data
-                         'size_total': size_total,
-                         'size_available': size_available,
-                         'uuid': uuid,
-                         })
+
+            if fstype == 'none':
+                continue
+
+            size_total, size_available = self._get_mount_size_facts(mount)
+
+            if mount in bind_mounts:
+                # only add if not already there, we might have a plain /etc/mtab
+                if not self.MTAB_BIND_MOUNT_RE.match(options):
+                    options += ",bind"
+
+            mount_info = {'mount': mount,
+                          'device': device,
+                          'fstype': fstype,
+                          'options': options,
+                          # statvfs data
+                          'size_total': size_total,
+                          'size_available': size_available,
+                          'uuid': uuids.get(device, 'N/A')}
+
+            mounts.append(mount_info)
+
+        self.facts['mounts'] = mounts
 
     def get_holders(self, block_dev_dict, sysdir):
         block_dev_dict['holders'] = []
@@ -1709,8 +1780,10 @@ def get_dmi_facts(self):
             else:
                 self.facts[k] = 'NA'
 
+
 class DragonFlyHardware(FreeBSDHardware):
-    pass
+    platform = 'DragonFly'
+
 
 class NetBSDHardware(Hardware):
     """
@@ -2068,7 +2141,10 @@ def __new__(cls, *arguments, **keyword):
         for sc in get_all_subclasses(Network):
             if sc.platform == platform.system():
                 subclass = sc
-        return super(cls, subclass).__new__(subclass, *arguments, **keyword)
+        if PY3:
+            return super(cls, subclass).__new__(subclass)
+        else:
+            return super(cls, subclass).__new__(subclass, *arguments, **keyword)
 
     def populate(self):
         return self.facts
@@ -2619,6 +2695,7 @@ class FreeBSDNetwork(GenericBsdIfconfigNetwork):
     """
     platform = 'FreeBSD'
 
+
 class DragonFlyNetwork(GenericBsdIfconfigNetwork):
     """
     This is the DragonFly Network Class.
@@ -2626,6 +2703,7 @@ class DragonFlyNetwork(GenericBsdIfconfigNetwork):
     """
     platform = 'DragonFly'
 
+
 class AIXNetwork(GenericBsdIfconfigNetwork):
     """
     This is the AIX Network Class.
@@ -2858,7 +2936,11 @@ def __new__(cls, *arguments, **keyword):
         for sc in get_all_subclasses(Virtual):
             if sc.platform == platform.system():
                 subclass = sc
-        return super(cls, subclass).__new__(subclass, *arguments, **keyword)
+
+        if PY3:
+            return super(cls, subclass).__new__(subclass)
+        else:
+            return super(cls, subclass).__new__(subclass, *arguments, **keyword)
 
     def populate(self):
         return self.facts
@@ -3055,7 +3137,7 @@ def get_virtual_facts(self):
         self.facts['virtualization_role'] = ''
 
 class DragonFlyVirtual(FreeBSDVirtual):
-    pass
+    platform = 'DragonFly'
 
 class OpenBSDVirtual(Virtual):
     """


Test patch:
diff --git a/test/units/module_utils/test_facts.py b/test/units/module_utils/test_facts.py
new file mode 100644
--- /dev/null
+++ b/test/units/module_utils/test_facts.py
@@ -0,0 +1,551 @@
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+# Make coding more python3-ish
+from __future__ import (absolute_import, division)
+__metaclass__ = type
+
+# for testing
+from ansible.compat.tests import unittest
+from ansible.compat.tests.mock import Mock, patch
+
+from ansible.module_utils import facts
+
+
+class BaseTestFactsPlatform(unittest.TestCase):
+    platform_id = 'Generic'
+    fact_class = facts.Hardware
+
+    """Verify that the automagic in Hardware.__new__ selects the right subclass."""
+    @patch('platform.system')
+    def test_new(self, mock_platform):
+        mock_platform.return_value = self.platform_id
+        inst = self.fact_class(module=Mock(), load_on_init=False)
+        self.assertIsInstance(inst, self.fact_class)
+        self.assertEqual(inst.platform, self.platform_id)
+
+    def test_subclass(self):
+        # 'Generic' will try to map to platform.system() that we are not mocking here
+        if self.platform_id == 'Generic':
+            return
+        inst = self.fact_class(module=Mock(), load_on_init=False)
+        self.assertIsInstance(inst, self.fact_class)
+        self.assertEqual(inst.platform, self.platform_id)
+
+
+class TestLinuxFactsPlatform(BaseTestFactsPlatform):
+    platform_id = 'Linux'
+    fact_class = facts.LinuxHardware
+
+
+class TestSunOSHardware(BaseTestFactsPlatform):
+    platform_id = 'SunOS'
+    fact_class = facts.SunOSHardware
+
+
+class TestOpenBSDHardware(BaseTestFactsPlatform):
+    platform_id = 'OpenBSD'
+    fact_class = facts.OpenBSDHardware
+
+
+class TestFreeBSDHardware(BaseTestFactsPlatform):
+    platform_id = 'FreeBSD'
+    fact_class = facts.FreeBSDHardware
+
+
+class TestDragonFlyHardware(BaseTestFactsPlatform):
+    platform_id = 'DragonFly'
+    fact_class = facts.DragonFlyHardware
+
+
+class TestNetBSDHardware(BaseTestFactsPlatform):
+    platform_id = 'NetBSD'
+    fact_class = facts.NetBSDHardware
+
+
+class TestAIXHardware(BaseTestFactsPlatform):
+    platform_id = 'AIX'
+    fact_class = facts.AIX
+
+
+class TestHPUXHardware(BaseTestFactsPlatform):
+    platform_id = 'HP-UX'
+    fact_class = facts.HPUX
+
+
+class TestDarwinHardware(BaseTestFactsPlatform):
+    platform_id = 'Darwin'
+    fact_class = facts.Darwin
+
+
+class TestGenericNetwork(BaseTestFactsPlatform):
+    platform_id = 'Generic'
+    fact_class = facts.Network
+
+
+class TestLinuxNetwork(BaseTestFactsPlatform):
+    platform_id = 'Generic'
+    fact_class = facts.Network
+
+
+class TestGenericBsdIfconfigNetwork(BaseTestFactsPlatform):
+    platform_id = 'Generic_BSD_Ifconfig'
+    fact_class = facts.GenericBsdIfconfigNetwork
+
+
+class TestHPUXNetwork(BaseTestFactsPlatform):
+    platform_id = 'HP-UX'
+    fact_class = facts.HPUXNetwork
+
+
+class TestDarwinNetwork(BaseTestFactsPlatform):
+    platform_id = 'Darwin'
+    fact_class = facts.DarwinNetwork
+
+
+class TestFreeBSDNetwork(BaseTestFactsPlatform):
+    platform_id = 'FreeBSD'
+    fact_class = facts.FreeBSDNetwork
+
+
+class TestDragonFlyNetwork(BaseTestFactsPlatform):
+    platform_id = 'DragonFly'
+    fact_class = facts.DragonFlyNetwork
+
+
+class TestAIXNetwork(BaseTestFactsPlatform):
+    platform_id = 'AIX'
+    fact_class = facts.AIXNetwork
+
+
+class TestOpenBSDNetwork(BaseTestFactsPlatform):
+    platform_id = 'OpenBSD'
+    fact_class = facts.OpenBSDNetwork
+
+
+class TestSunOSNetwork(BaseTestFactsPlatform):
+    platform_id = 'SunOS'
+    fact_class = facts.SunOSNetwork
+
+
+class TestLinuxVirtual(BaseTestFactsPlatform):
+    platform_id = 'Linux'
+    fact_class = facts.LinuxVirtual
+
+
+class TestFreeBSDVirtual(BaseTestFactsPlatform):
+    platform_id = 'FreeBSD'
+    fact_class = facts.FreeBSDNetwork
+
+
+class TestDragonFlyVirtual(BaseTestFactsPlatform):
+    platform_id = 'DragonFly'
+    fact_class = facts.DragonFlyNetwork
+
+
+class TestOpenBSDVirtual(BaseTestFactsPlatform):
+    platform_id = 'OpenBSD'
+    fact_class = facts.OpenBSDVirtual
+
+
+class TestHPUXVirtual(BaseTestFactsPlatform):
+    platform_id = 'HP-UX'
+    fact_class = facts.HPUXVirtual
+
+
+class TestSunOSVirtual(BaseTestFactsPlatform):
+    platform_id = 'SunOS'
+    fact_class = facts.SunOSVirtual
+
+
+LSBLK_OUTPUT = """
+/dev/sda
+/dev/sda1                             32caaec3-ef40-4691-a3b6-438c3f9bc1c0
+/dev/sda2                             66Ojcd-ULtu-1cZa-Tywo-mx0d-RF4O-ysA9jK
+/dev/mapper/fedora_dhcp129--186-swap  eae6059d-2fbe-4d1c-920d-a80bbeb1ac6d
+/dev/mapper/fedora_dhcp129--186-root  d34cf5e3-3449-4a6c-8179-a1feb2bca6ce
+/dev/mapper/fedora_dhcp129--186-home  2d3e4853-fa69-4ccf-8a6a-77b05ab0a42d
+/dev/sr0
+/dev/loop0                            0f031512-ab15-497d-9abd-3a512b4a9390
+/dev/loop1                            7c1b0f30-cf34-459f-9a70-2612f82b870a
+/dev/loop9                            0f031512-ab15-497d-9abd-3a512b4a9390
+/dev/loop9                            7c1b4444-cf34-459f-9a70-2612f82b870a
+/dev/mapper/docker-253:1-1050967-pool
+/dev/loop2
+/dev/mapper/docker-253:1-1050967-pool
+"""
+
+LSBLK_OUTPUT_2  = """
+/dev/sda
+/dev/sda1                            32caaec3-ef40-4691-a3b6-438c3f9bc1c0
+/dev/sda2                            66Ojcd-ULtu-1cZa-Tywo-mx0d-RF4O-ysA9jK
+/dev/mapper/fedora_dhcp129--186-swap eae6059d-2fbe-4d1c-920d-a80bbeb1ac6d
+/dev/mapper/fedora_dhcp129--186-root d34cf5e3-3449-4a6c-8179-a1feb2bca6ce
+/dev/mapper/fedora_dhcp129--186-home 2d3e4853-fa69-4ccf-8a6a-77b05ab0a42d
+/dev/mapper/an-example-mapper with a space in the name 84639acb-013f-4d2f-9392-526a572b4373
+/dev/sr0
+/dev/loop0                           0f031512-ab15-497d-9abd-3a512b4a9390
+"""
+
+LSBLK_UUIDS = {'/dev/sda1': '66Ojcd-ULtu-1cZa-Tywo-mx0d-RF4O-ysA9jK'}
+
+MTAB = """
+sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
+proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
+devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=8044400k,nr_inodes=2011100,mode=755 0 0
+securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
+tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0
+devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
+tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
+tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0
+cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
+pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0
+cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
+cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
+cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
+cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
+cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
+cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
+cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
+cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
+cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
+cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
+configfs /sys/kernel/config configfs rw,relatime 0 0
+/dev/mapper/fedora_dhcp129--186-root / ext4 rw,seclabel,relatime,data=ordered 0 0
+selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0
+systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=24,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
+debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0
+hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime 0 0
+tmpfs /tmp tmpfs rw,seclabel 0 0
+mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0
+/dev/loop0 /var/lib/machines btrfs rw,seclabel,relatime,space_cache,subvolid=5,subvol=/ 0 0
+/dev/sda1 /boot ext4 rw,seclabel,relatime,data=ordered 0 0
+/dev/mapper/fedora_dhcp129--186-home /home ext4 rw,seclabel,relatime,data=ordered 0 0
+tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=1611044k,mode=700,uid=1000,gid=1000 0 0
+gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
+fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
+"""
+
+MTAB_ENTRIES = \
+    [
+        ['sysfs',
+         '/sys',
+         'sysfs',
+         'rw,seclabel,nosuid,nodev,noexec,relatime',
+         '0',
+         '0'],
+        ['proc', '/proc', 'proc', 'rw,nosuid,nodev,noexec,relatime', '0', '0'],
+        ['devtmpfs',
+         '/dev',
+         'devtmpfs',
+         'rw,seclabel,nosuid,size=8044400k,nr_inodes=2011100,mode=755',
+         '0',
+         '0'],
+        ['securityfs',
+         '/sys/kernel/security',
+         'securityfs',
+         'rw,nosuid,nodev,noexec,relatime',
+         '0',
+         '0'],
+        ['tmpfs', '/dev/shm', 'tmpfs', 'rw,seclabel,nosuid,nodev', '0', '0'],
+        ['devpts',
+         '/dev/pts',
+         'devpts',
+         'rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000',
+         '0',
+         '0'],
+        ['tmpfs', '/run', 'tmpfs', 'rw,seclabel,nosuid,nodev,mode=755', '0', '0'],
+        ['tmpfs',
+         '/sys/fs/cgroup',
+         'tmpfs',
+         'ro,seclabel,nosuid,nodev,noexec,mode=755',
+         '0',
+         '0'],
+        ['cgroup',
+         '/sys/fs/cgroup/systemd',
+         'cgroup',
+         'rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd',
+         '0',
+         '0'],
+        ['pstore',
+         '/sys/fs/pstore',
+         'pstore',
+         'rw,seclabel,nosuid,nodev,noexec,relatime',
+         '0',
+         '0'],
+        ['cgroup',
+         '/sys/fs/cgroup/devices',
+         'cgroup',
+         'rw,nosuid,nodev,noexec,relatime,devices',
+         '0',
+         '0'],
+        ['cgroup',
+        '/sys/fs/cgroup/freezer',
+        'cgroup',
+        'rw,nosuid,nodev,noexec,relatime,freezer',
+        '0',
+        '0'],
+        ['cgroup',
+        '/sys/fs/cgroup/memory',
+        'cgroup',
+        'rw,nosuid,nodev,noexec,relatime,memory',
+        '0',
+        '0'],
+        ['cgroup',
+        '/sys/fs/cgroup/pids',
+        'cgroup',
+        'rw,nosuid,nodev,noexec,relatime,pids',
+        '0',
+        '0'],
+        ['cgroup',
+        '/sys/fs/cgroup/blkio',
+        'cgroup',
+        'rw,nosuid,nodev,noexec,relatime,blkio',
+        '0',
+        '0'],
+        ['cgroup',
+        '/sys/fs/cgroup/cpuset',
+        'cgroup',
+        'rw,nosuid,nodev,noexec,relatime,cpuset',
+        '0',
+        '0'],
+        ['cgroup',
+        '/sys/fs/cgroup/cpu,cpuacct',
+        'cgroup',
+        'rw,nosuid,nodev,noexec,relatime,cpu,cpuacct',
+        '0',
+        '0'],
+        ['cgroup',
+        '/sys/fs/cgroup/hugetlb',
+        'cgroup',
+        'rw,nosuid,nodev,noexec,relatime,hugetlb',
+        '0',
+        '0'],
+        ['cgroup',
+        '/sys/fs/cgroup/perf_event',
+        'cgroup',
+        'rw,nosuid,nodev,noexec,relatime,perf_event',
+        '0',
+        '0'],
+        ['cgroup',
+        '/sys/fs/cgroup/net_cls,net_prio',
+        'cgroup',
+        'rw,nosuid,nodev,noexec,relatime,net_cls,net_prio',
+        '0',
+        '0'],
+        ['configfs', '/sys/kernel/config', 'configfs', 'rw,relatime', '0', '0'],
+        ['/dev/mapper/fedora_dhcp129--186-root',
+        '/',
+        'ext4',
+        'rw,seclabel,relatime,data=ordered',
+        '0',
+        '0'],
+        ['selinuxfs', '/sys/fs/selinux', 'selinuxfs', 'rw,relatime', '0', '0'],
+        ['systemd-1',
+        '/proc/sys/fs/binfmt_misc',
+        'autofs',
+        'rw,relatime,fd=24,pgrp=1,timeout=0,minproto=5,maxproto=5,direct',
+        '0',
+        '0'],
+        ['debugfs', '/sys/kernel/debug', 'debugfs', 'rw,seclabel,relatime', '0', '0'],
+        ['hugetlbfs',
+        '/dev/hugepages',
+        'hugetlbfs',
+        'rw,seclabel,relatime',
+        '0',
+        '0'],
+        ['tmpfs', '/tmp', 'tmpfs', 'rw,seclabel', '0', '0'],
+        ['mqueue', '/dev/mqueue', 'mqueue', 'rw,seclabel,relatime', '0', '0'],
+        ['/dev/loop0',
+        '/var/lib/machines',
+        'btrfs',
+        'rw,seclabel,relatime,space_cache,subvolid=5,subvol=/',
+        '0',
+        '0'],
+        ['/dev/sda1', '/boot', 'ext4', 'rw,seclabel,relatime,data=ordered', '0', '0'],
+        # A 'none' fstype
+        ['/dev/sdz3', '/not/a/real/device', 'none', 'rw,seclabel,relatime,data=ordered', '0', '0'],
+        # lets assume this is a bindmount
+        ['/dev/sdz4', '/not/a/real/bind_mount', 'ext4', 'rw,seclabel,relatime,data=ordered', '0', '0'],
+        ['/dev/mapper/fedora_dhcp129--186-home',
+        '/home',
+        'ext4',
+        'rw,seclabel,relatime,data=ordered',
+        '0',
+        '0'],
+        ['tmpfs',
+        '/run/user/1000',
+        'tmpfs',
+        'rw,seclabel,nosuid,nodev,relatime,size=1611044k,mode=700,uid=1000,gid=1000',
+        '0',
+        '0'],
+        ['gvfsd-fuse',
+        '/run/user/1000/gvfs',
+        'fuse.gvfsd-fuse',
+        'rw,nosuid,nodev,relatime,user_id=1000,group_id=1000',
+        '0',
+        '0'],
+        ['fusectl', '/sys/fs/fuse/connections', 'fusectl', 'rw,relatime', '0', '0']]
+
+BIND_MOUNTS = ['/not/a/real/bind_mount']
+
+FINDMNT_OUTPUT = """
+/sys                            sysfs                                sysfs           rw,nosuid,nodev,noexec,relatime,seclabel
+/proc                           proc                                 proc            rw,nosuid,nodev,noexec,relatime
+/dev                            devtmpfs                             devtmpfs        rw,nosuid,seclabel,size=8044400k,nr_inodes=2011100,mode=755
+/sys/kernel/security            securityfs                           securityfs      rw,nosuid,nodev,noexec,relatime
+/dev/shm                        tmpfs                                tmpfs           rw,nosuid,nodev,seclabel
+/dev/pts                        devpts                               devpts          rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000
+/run                            tmpfs                                tmpfs           rw,nosuid,nodev,seclabel,mode=755
+/sys/fs/cgroup                  tmpfs                                tmpfs           ro,nosuid,nodev,noexec,seclabel,mode=755
+/sys/fs/cgroup/systemd          cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
+/sys/fs/pstore                  pstore                               pstore          rw,nosuid,nodev,noexec,relatime,seclabel
+/sys/fs/cgroup/devices          cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,devices
+/sys/fs/cgroup/freezer          cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,freezer
+/sys/fs/cgroup/memory           cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,memory
+/sys/fs/cgroup/pids             cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,pids
+/sys/fs/cgroup/blkio            cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,blkio
+/sys/fs/cgroup/cpuset           cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,cpuset
+/sys/fs/cgroup/cpu,cpuacct      cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
+/sys/fs/cgroup/hugetlb          cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,hugetlb
+/sys/fs/cgroup/perf_event       cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,perf_event
+/sys/fs/cgroup/net_cls,net_prio cgroup                               cgroup          rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
+/sys/kernel/config              configfs                             configfs        rw,relatime
+/                               /dev/mapper/fedora_dhcp129--186-root ext4            rw,relatime,seclabel,data=ordered
+/sys/fs/selinux                 selinuxfs                            selinuxfs       rw,relatime
+/proc/sys/fs/binfmt_misc        systemd-1                            autofs          rw,relatime,fd=24,pgrp=1,timeout=0,minproto=5,maxproto=5,direct
+/sys/kernel/debug               debugfs                              debugfs         rw,relatime,seclabel
+/dev/hugepages                  hugetlbfs                            hugetlbfs       rw,relatime,seclabel
+/tmp                            tmpfs                                tmpfs           rw,seclabel
+/dev/mqueue                     mqueue                               mqueue          rw,relatime,seclabel
+/var/lib/machines               /dev/loop0                           btrfs           rw,relatime,seclabel,space_cache,subvolid=5,subvol=/
+/boot                           /dev/sda1                            ext4            rw,relatime,seclabel,data=ordered
+/home                           /dev/mapper/fedora_dhcp129--186-home ext4            rw,relatime,seclabel,data=ordered
+/run/user/1000                  tmpfs                                tmpfs           rw,nosuid,nodev,relatime,seclabel,size=1611044k,mode=700,uid=1000,gid=1000
+/run/user/1000/gvfs             gvfsd-fuse                           fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
+/sys/fs/fuse/connections        fusectl                              fusectl         rw,relatime
+/not/a/real/bind_mount          /dev/sdz4[/some/other/path]     ext4    rw,relatime,seclabel,data=ordered
+"""
+
+
+class TestFactsLinuxHardwareGetMountFacts(unittest.TestCase):
+
+    # FIXME: mock.patch instead
+    def setUp(self):
+        # The @timeout tracebacks if there isn't a GATHER_TIMEOUT is None (the default until get_all_facts sets it via global)
+        facts.GATHER_TIMEOUT = 10
+
+    def tearDown(self):
+        facts.GATHER_TIMEOUT = None
+
+    # The Hardware subclasses freakout if instaniated directly, so
+    # mock platform.system and inst Hardware() so we get a LinuxHardware()
+    # we can test.
+    @patch('ansible.module_utils.facts.LinuxHardware._mtab_entries', return_value=MTAB_ENTRIES)
+    @patch('ansible.module_utils.facts.LinuxHardware._find_bind_mounts', return_value=BIND_MOUNTS)
+    @patch('ansible.module_utils.facts.LinuxHardware._lsblk_uuid', return_value=LSBLK_UUIDS)
+    def test_get_mount_facts(self,
+                             mock_lsblk_uuid,
+                             mock_find_bind_mounts,
+                             mock_mtab_entries):
+        module = Mock()
+        # Returns a LinuxHardware-ish
+        lh = facts.LinuxHardware(module=module, load_on_init=False)
+
+        # Nothing returned, just self.facts modified as a side effect
+        lh.get_mount_facts()
+        self.assertIsInstance(lh.facts, dict)
+        self.assertIn('mounts', lh.facts)
+        self.assertIsInstance(lh.facts['mounts'], list)
+        self.assertIsInstance(lh.facts['mounts'][0], dict)
+
+    @patch('ansible.module_utils.facts.get_file_content', return_value=MTAB)
+    def test_get_mtab_entries(self, mock_get_file_content):
+
+        module = Mock()
+        lh = facts.LinuxHardware(module=module, load_on_init=False)
+        mtab_entries = lh._mtab_entries()
+        self.assertIsInstance(mtab_entries, list)
+        self.assertIsInstance(mtab_entries[0], list)
+        self.assertEqual(len(mtab_entries), 34)
+
+    @patch('ansible.module_utils.facts.LinuxHardware._run_findmnt', return_value=(0, FINDMNT_OUTPUT,''))
+    def test_find_bind_mounts(self, mock_run_findmnt):
+        module = Mock()
+        lh = facts.LinuxHardware(module=module, load_on_init=False)
+        bind_mounts = lh._find_bind_mounts()
+
+        # If bind_mounts becomes another seq type, feel free to change
+        self.assertIsInstance(bind_mounts, set)
+        self.assertEqual(len(bind_mounts), 1)
+        self.assertIn('/not/a/real/bind_mount', bind_mounts)
+
+    @patch('ansible.module_utils.facts.LinuxHardware._run_findmnt', return_value=(37, '',''))
+    def test_find_bind_mounts_non_zero(self, mock_run_findmnt):
+        module = Mock()
+        lh = facts.LinuxHardware(module=module, load_on_init=False)
+        bind_mounts = lh._find_bind_mounts()
+
+        self.assertIsInstance(bind_mounts, set)
+        self.assertEqual(len(bind_mounts), 0)
+
+    def test_find_bind_mounts_no_findmnts(self):
+        module = Mock()
+        module.get_bin_path = Mock(return_value=None)
+        lh = facts.LinuxHardware(module=module, load_on_init=False)
+        bind_mounts = lh._find_bind_mounts()
+
+        self.assertIsInstance(bind_mounts, set)
+        self.assertEqual(len(bind_mounts), 0)
+
+    @patch('ansible.module_utils.facts.LinuxHardware._run_lsblk', return_value=(0, LSBLK_OUTPUT,''))
+    def test_lsblk_uuid(self, mock_run_lsblk):
+        module = Mock()
+        lh = facts.LinuxHardware(module=module, load_on_init=False)
+        lsblk_uuids = lh._lsblk_uuid()
+
+        self.assertIsInstance(lsblk_uuids, dict)
+        self.assertIn('/dev/loop9', lsblk_uuids)
+        self.assertIn('/dev/sda1', lsblk_uuids)
+        self.assertEquals(lsblk_uuids['/dev/sda1'], '32caaec3-ef40-4691-a3b6-438c3f9bc1c0')
+
+    @patch('ansible.module_utils.facts.LinuxHardware._run_lsblk', return_value=(37, LSBLK_OUTPUT,''))
+    def test_lsblk_uuid_non_zero(self, mock_run_lsblk):
+        module = Mock()
+        lh = facts.LinuxHardware(module=module, load_on_init=False)
+        lsblk_uuids = lh._lsblk_uuid()
+
+        self.assertIsInstance(lsblk_uuids, dict)
+        self.assertEquals(len(lsblk_uuids), 0)
+
+    def test_lsblk_uuid_no_lsblk(self):
+        module = Mock()
+        module.get_bin_path = Mock(return_value=None)
+        lh = facts.LinuxHardware(module=module, load_on_init=False)
+        lsblk_uuids = lh._lsblk_uuid()
+
+        self.assertIsInstance(lsblk_uuids, dict)
+        self.assertEquals(len(lsblk_uuids), 0)
+
+    @patch('ansible.module_utils.facts.LinuxHardware._run_lsblk', return_value=(0, LSBLK_OUTPUT_2,''))
+    def test_lsblk_uuid_dev_with_space_in_name(self, mock_run_lsblk):
+        module = Mock()
+        lh = facts.LinuxHardware(module=module, load_on_init=False)
+        lsblk_uuids = lh._lsblk_uuid()
+        self.assertIsInstance(lsblk_uuids, dict)
+        self.assertIn('/dev/loop0', lsblk_uuids)
+        self.assertIn('/dev/sda1', lsblk_uuids)
+        self.assertEquals(lsblk_uuids['/dev/mapper/an-example-mapper with a space in the name'], '84639acb-013f-4d2f-9392-526a572b4373')
+        self.assertEquals(lsblk_uuids['/dev/sda1'], '32caaec3-ef40-4691-a3b6-438c3f9bc1c0')