Repository: ansible/ansible-modules-extras

Issue:
Use -z when invoking pkg_add to allow installation of alternatives
##### Issue Type:

Bug Report/Feature Request
(WAS https://github.com/ansible/ansible/issues/8990)
##### Ansible Version:

1.6.1, but same in master
##### Environment:

OpenBSD
##### Summary:

openbsd_pkg 's module doesnt allow to install a package when it has alternatives - ie for example php-fpm, which could be 5.3 or 5.4 - or python, where 2.6, 2.7 and 3.3 are available
##### Steps To Reproduce:

playbook snippet:

```
- name: install php-fpm
  openbsd_pkg: name=php-fpm-5.4 state=installed
```
##### Expected Results:

php-fpm 5.4 should have installed
##### Actual Results:

```
failed: [twilight.rhaalovely.net] => {"failed": true, "item": ""}
msg: Error from http://ftp.fr.openbsd.org/pub/OpenBSD/5.5/packages/amd64/php-fpm-5.4.tgz
ftp: Error retrieving file: 404 Not Found
```

pkg_add has this feature that if you use -z, it will try fuzzy-matching the package name/version, if you provided it enough details

Various usescases for -z ..

```
$ sudo pkg_add -I php-fpm-5.4*  
Error from http://ftp.fr.openbsd.org/pub/OpenBSD/5.5/packages/amd64/php-fpm-5.4*.tgz
ftp: Error retrieving file: 404 Not Found
$ sudo pkg_add -Iz php-fpm      
Ambiguous: php-fpm could be php-fpm-5.3.28p3 php-fpm-5.4.24
$ sudo pkg_add -Iz php-fpm-5.4*
php-fpm-5.4.24: ok
```

If openbsd_pkg would add -z in pkg_add invocation in https://github.com/ansible/ansible/blob/devel/library/packaging/openbsd_pkg#L127, then i would be able to install the correct version of the package within the alternatives, specifying php-fpm-5.4\* in the name arg. 



Discussion:
Ok, the example i provided wasnt totally clear - after more testing on 5.5, 5.6 and -current, -z is needed for fuzzy matching but so far _only_ on the pX part of the pkgname (the patchlevel).

Ie in the case where the available packages are say php-fpm-5.3.29p2 php-fpm-5.5.18p1 php-fpm-5.4.34p0, pkg_add -Iz php-fpm-5.3 would fail (not precise enough for the tools) but pkg_add -Iz php-fpm-5.3.29 would install  php-fpm-5.3.29p2.

If the version specified is higher than any of the available, the latest one would be installed. Ie pkg_add -Iz php-fpm-5.6 would install php-fpm-5.5.18p1.

In any case, using -z would provide better matching than what we have now, where we need to specifiy exactly the version on the mirror.

Yes, this would be useful addition to the openbsd_pkg module in Ansible.

I have a task that fails now:

``` yaml
- name: Be sure nginx is installed
  openbsd_pkg: name=nginx-1.5.7 state=latest
```

Seems like `-z` would solve it:

```
openbsd% sudo pkg_add nginx
quirks-2.9 signed on 2014-07-31T22:37:55Z
Ambiguous: choose package for nginx
 a       0: <None>
         1: nginx-1.4.7p0
         2: nginx-1.4.7p0-lua
         3: nginx-1.4.7p0-naxsi
         4: nginx-1.4.7p0-passenger
         5: nginx-1.5.7p3
         6: nginx-1.5.7p3-lua
         7: nginx-1.5.7p3-naxsi
         8: nginx-1.5.7p3-passenger
Your choice: ^CFatal error: Caught SIGINT
 at /usr/libdata/perl5/OpenBSD/AddDelete.pm line 40.

openbsd% sudo pkg_add nginx-1.5.7
quirks-2.9 signed on 2014-07-31T22:37:55Z
Error from https://stable.mtier.org/updates/5.6/amd64/nginx-1.5.7.tgz
ftp: Error retrieving file: 404 Not Found
Error from http://ftp.su.se/pub/OpenBSD/5.6/packages/amd64/nginx-1.5.7.tgz
ftp: Error retrieving file: 404 Not Found
Can't find nginx-1.5.7
--- nginx-1.5.7 -------------------
Can't install nginx-1.5.7: not found

openbsd% sudo pkg_add -z nginx-1.5.7
quirks-2.9 signed on 2014-07-31T22:37:55Z
nginx-1.5.7p3: ok
The following new rcscripts were installed: /etc/rc.d/enginx
See rc.d(8) for details.
Look in /usr/local/share/doc/pkg-readmes for extra documentation.
```

@landryb Thanks for submitting this feature idea -- and thanks to @dentarg for adding on with additional info. Sorry it's taken us a while to get to triaging this. :)

@eest Pinging you to bring this feature request for your attention. Please note that you are under no particular obligation to add this feature.

We encourage all Ansible community members to submit pull requests to satisfy feature requests. Consult the module guidelines (http://docs.ansible.com/ansible/developing_modules.html) for more information. 

[module: packaging/os/openbsd_pkg.py]

I am against this change for the same reasons I was against it when we discussed it in https://github.com/ansible/ansible/issues/8990:
- pkg_delete has no support for -z. This means we would be installing packages that we can not delete using the same name.
- the pkg_add tool is pretty hard to interact with as is, because return codes and such differ depending on if the package contains version numbers etc. This can be seen in the code paths currently in the module where I have tried to document this. Throwing -z in the mix scares me from a complexity standpoint.

There is also another point I did not make last time:
- You can no longer be completely sure what package version is installed on your potentially large number of hosts.

I agree that it is unfortunate to sometimes need to be more specific regarding package names than one would like since it means more upkeep work is needed for playbooks. However, I feel that it is more important to have a strictly controlled environment when managing hosts in bulk.

I hope this makes sense to others.

If we do add this feature it should have a big warning stating what Patrik notes above.

This request has been in the back of my mind for a while. While I am still reluctant to add -z by default for the above reasons, I would probably be OK with adding a new argument to the module which activates it, something like "fuzzy=yes" perhaps. This would make it possible to have the module be as strict as it currently is by default, yet allow people to write less cumbersome playbooks if they want to.

What do people think about this?

That would just make our lifes easier imo. As for the previous points you make, we dont care if pkg_delete doesnt have -z, we can always get the exact pkgstem from another source (command pkg_info etc..) or gather the installed pkg list via facts and delete the correct version.

As for pkg_add being hard to interact with, we're always open to feedback on the pkg_tools, especially in improving return codes. 

> This request has been in the back of my mind for a while. While I am still reluctant to add -z by default for the above reasons, I would probably be OK with adding a new argument to the module which activates it, something like "fuzzy=yes" perhaps. This would make it possible to have the module be as strict as it currently is by default, yet allow people to write less cumbersome playbooks if they want to.
> 
> Sounds good to me!

So I have finally had some time to look into this and it is getting pretty complex right of the bat. The initial problem is this: There is currently a function called get_package_state() which is used to decide if the requested package is installed or not. This is the basis for making further decisions on what needs to be changed if anything. It depends on "pkg_info -e" to check for packages.

I am not sure what would be the correct way to convert a name given to "pkg_add -z" to "pkg_info -e", because the name handling of -z is pretty different. One of the examples given by @landryb is that installing a fuzzy version of "php-fpm-5.6" would pick up "php-fpm-5.5.18p1".

Any input on how this should be handled with pkg_info -e? It might be possible to just use the 'stem' part of the name with some sort of greater-than/less-than check for the version string, but it seems any such decision creates corner cases where the module will wrongly think nothing needs to be done.

@eest a quick thought, if possible, skip the pkg_info -e step and just execute pkg_add -z?

It might be a possible way forward but there are a few drawbacks:
- We would be hammering mirrors every time we are running a playbook which manages packages even if nothing needs to be done.
- The playbook will be slower because of this. I am not sure this is "worth it" for an operation that generally does not change anything except the very first time the playbook is run.

Just from quick testing: it took me about 3-10 seconds to run pkg_add -Iz when the package was already installed, and pkg_info of course returns instantly. This adds up when a playbook handles many packages.

If this is acceptable I guess we could build a package_present_fuzzy() function which does not care about the installed_state and just calls pkg_add -Iz(n) and parses the output from that command to decide if anything changed (or would change).

That would be okay for me, I think my use case so far only is "nginx", and I only manage a few OpenBSD servers, the situation might be different for others

Fwiw, an option is coming to pkg_add to select package names by 'branch', allowing one to do 'pkg_add autoconf%2.13 or php-fpm%5.6 ... which would solve this issue.

@landryb: Thank you for this information, while it might seem like this is a forgotten request I do mull it over from time to time. The combination of sharp corners of the implementation and what I perceive as a generally low request rate of the feature has stopped me from going forward with it currently.

If the pkg_add tool will be gaining functionality that solves the problem then I am more than happy about that. I did a quick glance of the cvs commits but found nothing related to pkg_add branches. I guess it is not yet available in -current?

Soon in -current, will be in 6.0.

@esst I have just found this issue, and I would love to have this resolved in ansible.
If you used the new pkg_add feature how would it provide support for current/older versions of openbsd?

@jody-frankowski: it would not. I am not very interested in adding a bunch of complexity only to cater to old systems. Basically I would rather not have the functionality at all rather than trying to deprecate it in the future. You will have to use specific package names for playbooks aimed at such older systems until replacing them instead.

It's in now in -current, cf http://anoncvs.spacehopper.org/openbsd-src/commit/?id=5cad1d0771d5798c01cac4962dae7c44f10b633b

@eest Well, I guess some versions could stay in production for long, but that sounds sensible to me.

@jody-frankowski - 'could stay in production' - remember that releases are only supported  for one year, so the problem will fix itself once 6.1 is released and 5.9 gets out of support... :)

@landryb I know, but it doesn't mean some wouldn't keep them in production ;)
To me, one of the problems with this is that some might not be encouraged to use Ansible, because a portion of their infrastructure can't be managed with Ansible (Even if It's their own fault).
Fortunately in our case as @eest said, they could still use specific package names.


Resolving patch:
diff --git a/packaging/os/openbsd_pkg.py b/packaging/os/openbsd_pkg.py
--- a/packaging/os/openbsd_pkg.py
+++ b/packaging/os/openbsd_pkg.py
@@ -19,10 +19,13 @@
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
+import platform
 import re
 import shlex
 import sqlite3
 
+from distutils.version import StrictVersion
+
 DOCUMENTATION = '''
 ---
 module: openbsd_pkg
@@ -82,6 +85,9 @@
 # Specify the default flavour to avoid ambiguity errors
 - openbsd_pkg: name=vim-- state=present
 
+# Specify a package branch (requires at least OpenBSD 6.0)
+- openbsd_pkg: name=python%3.5 state=present
+
 # Update all packages on the system
 - openbsd_pkg: name=* state=latest
 '''
@@ -94,47 +100,22 @@ def execute_command(cmd, module):
     cmd_args = shlex.split(cmd)
     return module.run_command(cmd_args)
 
-# Function used for getting the name of a currently installed package.
-def get_current_name(name, pkg_spec, module):
-    info_cmd = 'pkg_info'
-    (rc, stdout, stderr) = execute_command("%s" % (info_cmd), module)
-    if rc != 0:
-        return (rc, stdout, stderr)
-
-    if pkg_spec['version']:
-        pattern = "^%s" % name
-    elif pkg_spec['flavor']:
-        pattern = "^%s-.*-%s\s" % (pkg_spec['stem'], pkg_spec['flavor'])
-    else:
-        pattern = "^%s-" % pkg_spec['stem']
-
-    module.debug("get_current_name(): pattern = %s" % pattern)
-
-    for line in stdout.splitlines():
-        module.debug("get_current_name: line = %s" % line)
-        match = re.search(pattern, line)
-        if match:
-            current_name = line.split()[0]
-
-    return current_name
-
 # Function used to find out if a package is currently installed.
 def get_package_state(name, pkg_spec, module):
-    info_cmd = 'pkg_info -e'
+    info_cmd = 'pkg_info -Iq'
 
-    if pkg_spec['version']:
-        command = "%s %s" % (info_cmd, name)
-    elif pkg_spec['flavor']:
-        command = "%s %s-*-%s" % (info_cmd, pkg_spec['stem'], pkg_spec['flavor'])
-    else:
-        command = "%s %s-*" % (info_cmd, pkg_spec['stem'])
+    command = "%s inst:%s" % (info_cmd, name)
 
     rc, stdout, stderr = execute_command(command, module)
 
-    if (stderr):
+    if stderr:
         module.fail_json(msg="failed in get_package_state(): " + stderr)
 
-    if rc == 0:
+    if stdout:
+        # If the requested package name is just a stem, like "python", we may
+        # find multiple packages with that name.
+        pkg_spec['installed_names'] = [name for name in stdout.splitlines()]
+        module.debug("get_package_state(): installed_names = %s" % pkg_spec['installed_names'])
         return True
     else:
         return False
@@ -173,8 +154,14 @@ def package_present(name, installed_state, pkg_spec, module):
         # specific version is supplied or not.
         #
         # When a specific version is supplied the return code will be 0 when
-        # a package is found and 1 when it is not, if a version is not
-        # supplied the tool will exit 0 in both cases:
+        # a package is found and 1 when it is not. If a version is not
+        # supplied the tool will exit 0 in both cases.
+        #
+        # It is important to note that "version" relates to the
+        # packages-specs(7) notion of a version. If using the branch syntax
+        # (like "python%3.5") the version number is considered part of the
+        # stem, and the pkg_add behavior behaves the same as if the name did
+        # not contain a version (which it strictly speaking does not).
         if pkg_spec['version'] or build is True:
             # Depend on the return code.
             module.debug("package_present(): depending on return code")
@@ -231,25 +218,23 @@ def package_latest(name, installed_state, pkg_spec, module):
 
     if installed_state is True:
 
-        # Fetch name of currently installed package.
-        pre_upgrade_name = get_current_name(name, pkg_spec, module)
-
-        module.debug("package_latest(): pre_upgrade_name = %s" % pre_upgrade_name)
-
         # Attempt to upgrade the package.
         (rc, stdout, stderr) = execute_command("%s %s" % (upgrade_cmd, name), module)
 
         # Look for output looking something like "nmap-6.01->6.25: ok" to see if
         # something changed (or would have changed). Use \W to delimit the match
         # from progress meter output.
-        match = re.search("\W%s->.+: ok\W" % pre_upgrade_name, stdout)
-        if match:
-            if module.check_mode:
-                module.exit_json(changed=True)
-
-            changed = True
-        else:
-            changed = False
+        changed = False
+        for installed_name in pkg_spec['installed_names']:
+            module.debug("package_latest(): checking for pre-upgrade package name: %s" % installed_name)
+            match = re.search("\W%s->.+: ok\W" % installed_name, stdout)
+            if match:
+                module.debug("package_latest(): pre-upgrade package name match: %s" % installed_name)
+                if module.check_mode:
+                    module.exit_json(changed=True)
+
+                changed = True
+                break
 
         # FIXME: This part is problematic. Based on the issues mentioned (and
         # handled) in package_present() it is not safe to blindly trust stderr
@@ -301,7 +286,12 @@ def package_absent(name, installed_state, module):
 
 # Function used to parse the package name based on packages-specs(7).
 # The general name structure is "stem-version[-flavors]".
+#
+# Names containing "%" are a special variation not part of the
+# packages-specs(7) syntax. See pkg_add(1) on OpenBSD 6.0 or later for a
+# description.
 def parse_package_name(name, pkg_spec, module):
+    module.debug("parse_package_name(): parsing name: %s" % name)
     # Do some initial matches so we can base the more advanced regex on that.
     version_match = re.search("-[0-9]", name)
     versionless_match = re.search("--", name)
@@ -309,7 +299,7 @@ def parse_package_name(name, pkg_spec, module):
     # Stop if someone is giving us a name that both has a version and is
     # version-less at the same time.
     if version_match and versionless_match:
-        module.fail_json(msg="Package name both has a version and is version-less: " + name)
+        module.fail_json(msg="package name both has a version and is version-less: " + name)
 
     # If name includes a version.
     if version_match:
@@ -322,7 +312,7 @@ def parse_package_name(name, pkg_spec, module):
             pkg_spec['flavor']            = match.group('flavor')
             pkg_spec['style']             = 'version'
         else:
-            module.fail_json(msg="Unable to parse package name at version_match: " + name)
+            module.fail_json(msg="unable to parse package name at version_match: " + name)
 
     # If name includes no version but is version-less ("--").
     elif versionless_match:
@@ -335,7 +325,7 @@ def parse_package_name(name, pkg_spec, module):
             pkg_spec['flavor']            = match.group('flavor')
             pkg_spec['style']             = 'versionless'
         else:
-            module.fail_json(msg="Unable to parse package name at versionless_match: " + name)
+            module.fail_json(msg="unable to parse package name at versionless_match: " + name)
 
     # If name includes no version, and is not version-less, it is all a stem.
     else:
@@ -348,14 +338,27 @@ def parse_package_name(name, pkg_spec, module):
             pkg_spec['flavor']            = None
             pkg_spec['style']             = 'stem'
         else:
-            module.fail_json(msg="Unable to parse package name at else: " + name)
+            module.fail_json(msg="unable to parse package name at else: " + name)
+
+    # If the stem contains an "%" then it needs special treatment.
+    branch_match = re.search("%", pkg_spec['stem'])
+    if branch_match:
+
+        branch_release = "6.0"
+
+        if version_match or versionless_match:
+            module.fail_json(msg="package name using 'branch' syntax also has a version or is version-less: " + name)
+        if StrictVersion(platform.release()) < StrictVersion(branch_release):
+            module.fail_json(msg="package name using 'branch' syntax requires at least OpenBSD %s: %s" % (branch_release, name))
+
+        pkg_spec['style'] = 'branch'
 
     # Sanity check that there are no trailing dashes in flavor.
     # Try to stop strange stuff early so we can be strict later.
     if pkg_spec['flavor']:
         match = re.search("-$", pkg_spec['flavor'])
         if match:
-            module.fail_json(msg="Trailing dash in flavor: " + pkg_spec['flavor'])
+            module.fail_json(msg="trailing dash in flavor: " + pkg_spec['flavor'])
 
 # Function used for figuring out the port path.
 def get_package_source_path(name, pkg_spec, module):
@@ -364,9 +367,14 @@ def get_package_source_path(name, pkg_spec, module):
         return 'databases/sqlports'
     else:
         # try for an exact match first
-        conn = sqlite3.connect('/usr/local/share/sqlports')
+        sqlports_db_file = '/usr/local/share/sqlports'
+        if not os.path.isfile(sqlports_db_file):
+            module.fail_json(msg="sqlports file '%s' is missing" % sqlports_db_file)
+
+        conn = sqlite3.connect(sqlports_db_file)
         first_part_of_query = 'SELECT fullpkgpath, fullpkgname FROM ports WHERE fullpkgname'
         query = first_part_of_query + ' = ?'
+        module.debug("package_package_source_path(): exact query: %s" % query)
         cursor = conn.execute(query, (name,))
         results = cursor.fetchall()
 
@@ -376,11 +384,14 @@ def get_package_source_path(name, pkg_spec, module):
             query = first_part_of_query + ' LIKE ?'
             if pkg_spec['flavor']:
                 looking_for += pkg_spec['flavor_separator'] + pkg_spec['flavor']
+                module.debug("package_package_source_path(): fuzzy flavor query: %s" % query)
                 cursor = conn.execute(query, (looking_for,))
             elif pkg_spec['style'] == 'versionless':
                 query += ' AND fullpkgname NOT LIKE ?'
+                module.debug("package_package_source_path(): fuzzy versionless query: %s" % query)
                 cursor = conn.execute(query, (looking_for, "%s-%%" % looking_for,))
             else:
+                module.debug("package_package_source_path(): fuzzy query: %s" % query)
                 cursor = conn.execute(query, (looking_for,))
             results = cursor.fetchall()
 
@@ -464,8 +475,9 @@ def main():
         # build sqlports if its not installed yet
         pkg_spec = {}
         parse_package_name('sqlports', pkg_spec, module)
-        installed_state = get_package_state(name, pkg_spec, module)
+        installed_state = get_package_state('sqlports', pkg_spec, module)
         if not installed_state:
+            module.debug("main(): installing 'sqlports' because build=%s" % module.params['build'])
             package_present('sqlports', installed_state, pkg_spec, module)
 
     if name == '*':
@@ -479,6 +491,11 @@ def main():
         pkg_spec = {}
         parse_package_name(name, pkg_spec, module)
 
+        # Not sure how the branch syntax is supposed to play together
+        # with build mode. Disable it for now.
+        if pkg_spec['style'] == 'branch' and module.params['build'] is True:
+            module.fail_json(msg="the combination of 'branch' syntax and build=%s is not supported: %s" % (module.params['build'], name))
+
         # Get package state.
         installed_state = get_package_state(name, pkg_spec, module)
 


Test patch: