Repository: ansible/ansible

Issue:
consul_kv: putting string into Consul is not idempotent in Python 3
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and devel branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report

##### COMPONENT NAME
<!---
Name of the module, plugin, task or feature
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path
-->
consul_kv

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.2.0
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.2 (default, Sep 13 2017, 14:26:54) [GCC 4.9.2]
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
N/A

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
N/A

##### SUMMARY
<!--- Explain the problem briefly -->
Key values are considered by the `consul_kv` module as strings. In Python 2 we `put` these (byte) `str` to `python-consul` and we `get` the same type of strings back. However, with Python 3, we `put` (utf-8) `str` in (which if you follow `python-consul` down, get sent off to Consul as utf-8 encoded) but we get `byte`s back. 

Python 2:
```python
>>> value
'Hello World'
>>> consul_client.kv.put(key, value)
True
>>> consul_client.kv.get(key, token=token)[1]["Value"]
'Hello World'
>>> consul_client.kv.get(key)[1]["Value"] == value
True
```

Python 3:
```python
>>> value
'Hello World'
>>> consul_client.kv.put(key, value)
True
>>> consul_client.kv.get(key)[1]["Value"]
b'Hello World'
>>> consul_client.kv.get(key)[1]["Value"] == value
False
```

In Python 3 a string is not equal to a sequence of bytes, which leads to the equality checks in the `consul_kv` module always determining that the value stored in Consul is not the one that is desired:
https://github.com/ansible/ansible/blob/a1d60741a7426824cf5eaa933d7d686c470843b1/lib/ansible/modules/clustering/consul_kv.py#L162
https://github.com/ansible/ansible/blob/a1d60741a7426824cf5eaa933d7d686c470843b1/lib/ansible/modules/clustering/consul_kv.py#L189


##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: Add or update the value associated with a key in the key/value store
  consul_kv:
    key: somekey
    value: somevalue
```

##### EXPECTED RESULTS
Idempotency.

##### ACTUAL RESULTS
Changes every time.



Discussion:
Files identified in the description:
* [lib/ansible/modules/clustering/consul_kv.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/clustering/consul_kv.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @sgargan
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/modules/clustering/consul_kv.py b/lib/ansible/modules/clustering/consul_kv.py
--- a/lib/ansible/modules/clustering/consul_kv.py
+++ b/lib/ansible/modules/clustering/consul_kv.py
@@ -1,6 +1,7 @@
 #!/usr/bin/python
 #
 # (c) 2015, Steve Gargan <steve.gargan@gmail.com>
+# (c) 2017, 2018 Genome Research Ltd.
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
 
 from __future__ import absolute_import, division, print_function
@@ -28,7 +29,8 @@
   - requests
 version_added: "2.0"
 author:
-- Steve Gargan (@sgargan)
+  - Steve Gargan (@sgargan)
+  - Colin Nolan (@colin-nolan)
 options:
     state:
         description:
@@ -122,6 +124,8 @@
     state: acquire
 '''
 
+from ansible.module_utils._text import to_text
+
 try:
     import consul
     from requests.exceptions import ConnectionError
@@ -132,6 +136,27 @@
 from ansible.module_utils.basic import AnsibleModule
 
 
+def _has_value_changed(consul_client, key, target_value):
+    """
+    Uses the given Consul client to determine if the value associated to the given key is different to the given target
+    value.
+    :param consul_client: Consul connected client
+    :param key: key in Consul
+    :param target_value: value to be associated to the key
+    :return: tuple where the first element is the value of the "X-Consul-Index" header and the second is `True` if the
+    value has changed (i.e. the stored value is not the target value)
+    """
+    index, existing = consul_client.kv.get(key)
+    if not existing:
+        return index, True
+    try:
+        changed = to_text(existing['Value'], errors='surrogate_or_strict') != target_value
+        return index, changed
+    except UnicodeError:
+        # Existing value was not decodable but all values we set are valid utf-8
+        return index, True
+
+
 def execute(module):
 
     state = module.params.get('state')
@@ -157,9 +182,8 @@ def lock(module, state):
             msg='%s of lock for %s requested but no session supplied' %
             (state, key))
 
-    index, existing = consul_api.kv.get(key)
+    index, changed = _has_value_changed(consul_api, key, value)
 
-    changed = not existing or (existing and existing['Value'] != value)
     if changed and not module.check_mode:
         if state == 'acquire':
             changed = consul_api.kv.put(key, value,
@@ -184,14 +208,14 @@ def add_value(module):
     key = module.params.get('key')
     value = module.params.get('value')
 
-    index, existing = consul_api.kv.get(key)
+    index, changed = _has_value_changed(consul_api, key, value)
 
-    changed = not existing or (existing and existing['Value'] != value)
     if changed and not module.check_mode:
         changed = consul_api.kv.put(key, value,
                                     cas=module.params.get('cas'),
                                     flags=module.params.get('flags'))
 
+    stored = None
     if module.params.get('retrieve'):
         index, stored = consul_api.kv.get(key)
 


Test patch: