Repository: ansible/ansible

Issue:
yum module parsing of wrapped 'yum check-update' output causes package updates to be skipped
##### ISSUE TYPE
Bug Report

##### COMPONENT NAME
yum module

##### ANSIBLE VERSION
2.2.1 (2.2.1.0-1ppa~xenial)

##### CONFIGURATION

##### OS / ENVIRONMENT
Control machine: Ubuntu 16.04
Remote machine: Centos 7.3.1611

##### SUMMARY
After fix this issue [#4318](https://github.com/ansible/ansible-modules-core/issues/4318)
the first packet (in alphabetical order output  of yum check-update),  that requires update, is not updated  -  not correctly parsed output of 'yum check-update'. The first line, which describes a new package and its version is added to the previous one and the package "excluded" from the update.

Problematic piece of [code](https://github.com/ansible/ansible-modules-core/blob/59c9a6d3c20e73641c430fdaa3f250c1ff431dcc/packaging/os/yum.py#L804):
` # remove incorrect new lines in longer columns in output from yum check-update
out=re.sub('\n\W+', ' ', out)`

workaround:
 this [patch:](https://gist.github.com/saamich/b701c1d45cf877ddc3938c5cb63aec30) work for i686/x86-64 and noarch packages. Rpm package name in "yum check-update" output ends to the architecture name under which the assembled package.

##### STEPS TO REPRODUCE
1) execute "yum check-update" on not updated node with centos 7.3.1611 (example - NODE) and look first package, which needs to be updated(example PKG-NAME).
2) execute ansible on this node 
ansibble NODE -m yum -a 'name=PKG-NAME state=latest'

##### EXPECTED RESULTS
package updated

##### ACTUAL RESULTS
package not updated


Discussion:
Could you provide an example of the 'yum check-update' output that is causing the problem?
hello, this output after "yum check-update 2>&1 >file"

[yum_check-update.txt](https://github.com/ansible/ansible/files/739507/yum_check-update.txt)

may be fix this issue (not full tested)
`out=re.sub(r'(\.[^\.\s]+)\W+(\w+)', r'\1 \2', out)`
My team ran into this issue as well when trying to install the latest ceph package with Ansible 2.2.1.  After troubleshooting the yum plugin we noticed it was lopping off the first package ceph.x86_64. I've included a link to the output of `yum check-update 2>&1 > file` as my output looks different than saamich's.

[https://github.com/digi691/check_update_output/blob/master/yum_check_update_ceph.txt](url)

Also like to add the the remote machine is also CentOS Linux release 7.3.1611.


Can confirm what seems like the same behaviour with the 'first' package being skipped when managing CentOS 6 and CentOS 7 hosts with Ansible 2.2.1.0.

I'm using Fedora 24 as the Ansible control machine, but the same occurs with CentOS 6/7. https://gist.github.com/mvermaes/3ac84bb4ac3a8060470545540ca52673 has an example of how to reproduce in a CentOS 6 Vagrant box.
Repository name to can also be transferred to the new line. In file output from yum check-update > file
[log.txt](https://github.com/ansible/ansible/files/747963/log.txt)

repoquery work fine
repoquery -q -a --qf="%{name} %{version}-%{release}.%{arch} %{repoid}" --pkgnarrow=updates
[log.txt](https://github.com/ansible/ansible/files/747968/log.txt)

workaround for use repoquery instead 'yum check-update' for 'state=latest'
[patch.txt](https://github.com/ansible/ansible/files/748239/patch.txt)


Resolving patch:
diff --git a/lib/ansible/modules/packaging/os/yum.py b/lib/ansible/modules/packaging/os/yum.py
--- a/lib/ansible/modules/packaging/os/yum.py
+++ b/lib/ansible/modules/packaging/os/yum.py
@@ -844,6 +844,53 @@ def remove(module, items, repoq, yum_basecmd, conf_file, en_repos, dis_repos, in
 
     return res
 
+
+def run_check_update(module, yum_basecmd):
+    # run check-update to see if we have packages pending
+    rc, out, err = module.run_command(yum_basecmd + ['check-update'])
+    return rc, out, err
+
+
+def parse_check_update(check_update_output):
+    updates = {}
+
+    # remove incorrect new lines in longer columns in output from yum check-update
+    # yum line wrapping can move the repo to the next line
+    #
+    # Meant to filter out sets of lines like:
+    #  some_looooooooooooooooooooooooooooooooooooong_package_name   1:1.2.3-1.el7
+    #                                                                    some-repo-label
+    #
+    # But it also needs to avoid catching lines like:
+    # Loading mirror speeds from cached hostfile
+    #
+    # ceph.x86_64                               1:11.2.0-0.el7                    ceph
+
+    # preprocess string and filter out empty lines so the regex below works
+    out = re.sub('\n[^\w]\W+(.*)', ' \1',
+                 check_update_output)
+
+    available_updates = out.split('\n')
+
+    # build update dictionary
+    for line in available_updates:
+        line = line.split()
+        # ignore irrelevant lines
+        # '*' in line matches lines like mirror lists:
+        #      * base: mirror.corbina.net
+        # len(line) != 3 could be junk or a continuation
+        #
+        # FIXME: what is  the '.' not in line  conditional for?
+
+        if '*' in line or len(line) != 3 or '.' not in line[0]:
+            continue
+        else:
+            pkg, version, repo = line
+            name, dist = pkg.rsplit('.', 1)
+            updates.update({name: {'version': version, 'dist': dist, 'repo': repo}})
+    return updates
+
+
 def latest(module, items, repoq, yum_basecmd, conf_file, en_repos, dis_repos, installroot='/'):
 
     res = {}
@@ -862,26 +909,13 @@ def latest(module, items, repoq, yum_basecmd, conf_file, en_repos, dis_repos, in
     if '*' in items:
         update_all = True
 
-    # run check-update to see if we have packages pending
-    rc, out, err = module.run_command(yum_basecmd + ['check-update'])
+    rc, out, err = run_check_update(module, yum_basecmd)
+
     if rc == 0 and update_all:
         res['results'].append('Nothing to do here, all packages are up to date')
         return res
     elif rc == 100:
-        # remove incorrect new lines in longer columns in output from yum check-update
-        out=re.sub('\n\W+', ' ', out)
-        available_updates = out.split('\n')
-        # build update dictionary
-        for line in available_updates:
-            line = line.split()
-            # ignore irrelevant lines
-            # FIXME... revisit for something less kludgy
-            if '*' in line or len(line) != 3 or '.' not in line[0]:
-                continue
-            else:
-                pkg, version, repo = line
-                name, dist = pkg.rsplit('.', 1)
-                updates.update({name: {'version': version, 'dist': dist, 'repo': repo}})
+        updates = parse_check_update(out)
     elif rc == 1:
         res['msg'] = err
         res['rc'] = rc
@@ -920,6 +954,7 @@ def latest(module, items, repoq, yum_basecmd, conf_file, en_repos, dis_repos, in
                     nothing_to_do = False
                     break
 
+
                 # this contains the full NVR and spec could contain wildcards
                 # or virtual provides (like "python-*" or "smtp-daemon") while
                 # updates contains name only.


Test patch:
diff --git a/test/units/modules/packaging/os/test_yum.py b/test/units/modules/packaging/os/test_yum.py
new file mode 100644
--- /dev/null
+++ b/test/units/modules/packaging/os/test_yum.py
@@ -0,0 +1,176 @@
+
+# -*- coding: utf-8 -*-
+from ansible.compat.tests import unittest
+
+from ansible.modules.packaging.os import yum
+
+
+yum_plugin_load_error = """
+Plugin "product-id" can't be imported
+Plugin "search-disabled-repos" can't be imported
+Plugin "subscription-manager" can't be imported
+Plugin "product-id" can't be imported
+Plugin "search-disabled-repos" can't be imported
+Plugin "subscription-manager" can't be imported
+"""
+
+# from https://github.com/ansible/ansible/issues/20608#issuecomment-276106505
+wrapped_output_1 = """
+Загружены модули: fastestmirror
+Loading mirror speeds from cached hostfile
+ * base: mirror.h1host.ru
+ * extras: mirror.h1host.ru
+ * updates: mirror.h1host.ru
+
+vms-agent.x86_64                            0.0-9                            dev
+"""
+
+# from https://github.com/ansible/ansible/issues/20608#issuecomment-276971275
+wrapped_output_2 = """
+Загружены модули: fastestmirror
+Loading mirror speeds from cached hostfile
+ * base: mirror.corbina.net
+ * extras: mirror.corbina.net
+ * updates: mirror.corbina.net
+
+empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty.x86_64
+                                    0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1-0
+                                                                         addons
+libtiff.x86_64                      4.0.3-27.el7_3                       updates
+"""
+
+# From https://github.com/ansible/ansible/issues/20608#issuecomment-276698431
+wrapped_output_3 = """
+Loaded plugins: fastestmirror, langpacks
+Loading mirror speeds from cached hostfile
+
+ceph.x86_64                               1:11.2.0-0.el7                    ceph
+ceph-base.x86_64                          1:11.2.0-0.el7                    ceph
+ceph-common.x86_64                        1:11.2.0-0.el7                    ceph
+ceph-mds.x86_64                           1:11.2.0-0.el7                    ceph
+ceph-mon.x86_64                           1:11.2.0-0.el7                    ceph
+ceph-osd.x86_64                           1:11.2.0-0.el7                    ceph
+ceph-selinux.x86_64                       1:11.2.0-0.el7                    ceph
+libcephfs1.x86_64                         1:11.0.2-0.el7                    ceph
+librados2.x86_64                          1:11.2.0-0.el7                    ceph
+libradosstriper1.x86_64                   1:11.2.0-0.el7                    ceph
+librbd1.x86_64                            1:11.2.0-0.el7                    ceph
+librgw2.x86_64                            1:11.2.0-0.el7                    ceph
+python-cephfs.x86_64                      1:11.2.0-0.el7                    ceph
+python-rados.x86_64                       1:11.2.0-0.el7                    ceph
+python-rbd.x86_64                         1:11.2.0-0.el7                    ceph
+"""
+
+# from https://github.com/ansible/ansible-modules-core/issues/4318#issuecomment-251416661
+wrapped_output_4 = """
+ipxe-roms-qemu.noarch                 20160127-1.git6366fa7a.el7
+                                                            rhelosp-9.0-director-puddle
+quota.x86_64                          1:4.01-11.el7_2.1     rhelosp-rhel-7.2-z
+quota-nls.noarch                      1:4.01-11.el7_2.1     rhelosp-rhel-7.2-z
+rdma.noarch                           7.2_4.1_rc6-2.el7     rhelosp-rhel-7.2-z
+screen.x86_64                         4.1.0-0.23.20120314git3c2946.el7_2
+                                                            rhelosp-rhel-7.2-z
+sos.noarch                            3.2-36.el7ost.2       rhelosp-9.0-puddle
+sssd-client.x86_64                    1.13.0-40.el7_2.12    rhelosp-rhel-7.2-z
+"""
+
+
+# A 'normal-ish' yum check-update output, without any wrapped lines
+unwrapped_output_rhel7 = """
+
+Loaded plugins: etckeeper, product-id, search-disabled-repos, subscription-
+              : manager
+This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
+
+NetworkManager-openvpn.x86_64         1:1.2.6-1.el7           epel
+NetworkManager-openvpn-gnome.x86_64   1:1.2.6-1.el7           epel
+cabal-install.x86_64                  1.16.1.0-2.el7          epel
+cgit.x86_64                           1.1-1.el7               epel
+python34-libs.x86_64                  3.4.5-3.el7             epel
+python34-test.x86_64                  3.4.5-3.el7             epel
+python34-tkinter.x86_64               3.4.5-3.el7             epel
+python34-tools.x86_64                 3.4.5-3.el7             epel
+qgit.x86_64                           2.6-4.el7               epel
+rdiff-backup.x86_64                   1.2.8-12.el7            epel
+stoken-libs.x86_64                    0.91-1.el7              epel
+xlockmore.x86_64                      5.49-2.el7              epel
+"""
+
+# Some wrapped obsoletes for prepending to output for testing both
+wrapped_output_rhel7_obsoletes_postfix = """
+Obsoleting Packages
+ddashboard.x86_64                     0.2.0.1-1.el7_3         mhlavink-developerdashboard
+    developerdashboard.x86_64         0.1.12.2-1.el7_2        @mhlavink-developerdashboard
+python-bugzilla.noarch                1.2.2-3.el7_2.1         mhlavink-developerdashboard
+    python-bugzilla-develdashboardfixes.noarch
+                                      1.2.2-3.el7             @mhlavink-developerdashboard
+python2-futures.noarch                3.0.5-1.el7             epel
+    python-futures.noarch             3.0.3-1.el7             @epel
+python2-pip.noarch                    8.1.2-5.el7             epel
+    python-pip.noarch                 7.1.0-1.el7             @epel
+python2-pyxdg.noarch                  0.25-6.el7              epel
+    pyxdg.noarch                      0.25-5.el7              @epel
+python2-simplejson.x86_64             3.10.0-1.el7            epel
+    python-simplejson.x86_64          3.3.3-1.el7             @epel
+Security: kernel-3.10.0-327.28.2.el7.x86_64 is an installed security update
+Security: kernel-3.10.0-327.22.2.el7.x86_64 is the currently running version
+"""
+
+unwrapped_output_rhel7_obsoletes = unwrapped_output_rhel7 + wrapped_output_rhel7_obsoletes_postfix
+unwrapped_output_rhel7_expected_pkgs = ["NetworkManager-openvpn", "NetworkManager-openvpn-gnome", "cabal-install",
+                                        "cgit", "python34-libs", "python34-test", "python34-tkinter",
+                                        "python34-tools", "qgit", "rdiff-backup", "stoken-libs", "xlockmore"]
+
+
+class TestYumUpdateCheckParse(unittest.TestCase):
+    def _assert_expected(self, expected_pkgs, result):
+
+        for expected_pkg in expected_pkgs:
+            self.assertIn(expected_pkg, result)
+        self.assertEqual(len(result), len(expected_pkgs))
+        self.assertIsInstance(result, dict)
+
+    def test_empty_output(self):
+        res = yum.parse_check_update("")
+        expected_pkgs = []
+        self._assert_expected(expected_pkgs, res)
+
+    def test_plugin_load_error(self):
+        res = yum.parse_check_update(yum_plugin_load_error)
+        expected_pkgs = []
+        self._assert_expected(expected_pkgs, res)
+
+    def test_wrapped_output_1(self):
+        res = yum.parse_check_update(wrapped_output_1)
+        expected_pkgs = ["vms-agent"]
+        self._assert_expected(expected_pkgs, res)
+
+    def test_wrapped_output_2(self):
+        res = yum.parse_check_update(wrapped_output_2)
+        expected_pkgs = ["empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty-empty",
+                         "libtiff"]
+
+        self._assert_expected(expected_pkgs, res)
+
+    def test_wrapped_output_3(self):
+        res = yum.parse_check_update(wrapped_output_3)
+        expected_pkgs = ["ceph", "ceph-base", "ceph-common", "ceph-mds",
+                         "ceph-mon", "ceph-osd", "ceph-selinux", "libcephfs1",
+                         "librados2", "libradosstriper1", "librbd1", "librgw2",
+                         "python-cephfs", "python-rados", "python-rbd"]
+        self._assert_expected(expected_pkgs, res)
+
+    def test_wrapped_output_4(self):
+        res = yum.parse_check_update(wrapped_output_4)
+
+        expected_pkgs = ["ipxe-roms-qemu", "quota", "quota-nls", "rdma", "screen",
+                         "sos", "sssd-client"]
+        self._assert_expected(expected_pkgs, res)
+
+    def test_wrapped_output_rhel7(self):
+        res = yum.parse_check_update(unwrapped_output_rhel7)
+        self._assert_expected(unwrapped_output_rhel7_expected_pkgs, res)
+
+    def test_wrapped_output_rhel7_obsoletes(self):
+        res = yum.parse_check_update(unwrapped_output_rhel7_obsoletes)
+        self._assert_expected(unwrapped_output_rhel7_expected_pkgs, res)