Repository: ansible-collections/community.general

Issue:
redis cache plugin does not accept (bracketed) IPv6 addresses
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Explain the problem briefly below -->
The redis cache plugin does not accept a fact_caching_connection with an IPv6 address


##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
redis cache plugin


##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0]
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
INTERPRETER_PYTHON(/etc/ansible/ansible.cfg) = auto_legacy_silent
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False
```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
I used Debian Buster 10.4 backports packages



##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->
Change CACHE_PLUGIN_CONNECTION(/etc/ansible/ansible.cfg) to a value with an IPv6 address, e.g. "[::1]:6379:0" (In my actual use case, its a link local unicast address from another machine, but ip6-localhost produces the same result) and use the redis fact cache plugin


<!--- Paste example playbooks or commands between quotes below -->
```yaml
- hosts: all
  tasks:
    - setup:
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->

The play would run


##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
ansible won't run at all


<!--- Paste verbatim command output between quotes -->
```paste below
ERROR! Unexpected Exception, this is probably a bug: invalid literal for int() with base 10: ''
the full traceback was:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 988, in get_connection
    connection = self._available_connections.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/ansible", line 123, in <module>
    exit_code = cli.run()
  File "/usr/lib/python3/dist-packages/ansible/cli/adhoc.py", line 165, in run
    result = self._tqm.run(play)
  File "/usr/lib/python3/dist-packages/ansible/executor/task_queue_manager.py", line 241, in run
    play_return = strategy.run(iterator, play_context)
  File "/usr/lib/python3/dist-packages/ansible/plugins/strategy/linear.py", line 282, in run
    _hosts=self._hosts_cache, _hosts_all=self._hosts_cache_all)
  File "/usr/lib/python3/dist-packages/ansible/vars/manager.py", line 305, in get_vars
    facts = wrap_var(self._fact_cache.get(host.name, {}))
  File "/usr/lib/python3.7/_collections_abc.py", line 660, in get
    return self[key]
  File "/usr/lib/python3/dist-packages/ansible/vars/fact_cache.py", line 31, in __getitem__
    if not self._plugin.contains(key):
  File "/usr/lib/python3/dist-packages/ansible/plugins/cache/redis.py", line 132, in contains
    self._expire_keys()
  File "/usr/lib/python3/dist-packages/ansible/plugins/cache/redis.py", line 125, in _expire_keys
    self._db.zremrangebyscore(self._keys_set, 0, expiry_age)
  File "/usr/lib/python3/dist-packages/redis/client.py", line 2532, in zremrangebyscore
    return self.execute_command('ZREMRANGEBYSCORE', name, min, max)
  File "/usr/lib/python3/dist-packages/redis/client.py", line 772, in execute_command
    connection = pool.get_connection(command_name, **options)
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 990, in get_connection
    connection = self.make_connection()
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 1025, in make_connection
    return self.connection_class(**self.connection_kwargs)
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 452, in __init__
    self.port = int(port)
ValueError: invalid literal for int() with base 10: ''

```



Discussion:
Files identified in the description:
* [`plugins/cache/redis.py`](https://github.com/ansible-collections/community.general/blob/master/plugins/cache/redis.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
i can't reproduce issue:
```
[mator@archlinux ansible.dev]$ cat ansible.cfg 
[defaults]
inventory = hosts
# gather_timeout = 20
# debug=true
log_path = ./log
ANSIBLE_NOCOWS = 1
interpreter_python = auto_silent
collections_paths = ./collections
pipelining = ANSIBLE_PIPELINING
DEVEL_WARNING = 0
fact_caching_connection = [::1]

[mator@archlinux ansible.dev]$ ansible-config dump --only-changed               
CACHE_PLUGIN_CONNECTION(/home/mator/ansible.dev/ansible.cfg) = [::1]
COLLECTIONS_PATHS(/home/mator/ansible.dev/ansible.cfg) = ['/home/mator/ansible.dev/collections']
DEFAULT_HOST_LIST(/home/mator/ansible.dev/ansible.cfg) = ['/home/mator/ansible.dev/hosts']
DEFAULT_LOG_PATH(/home/mator/ansible.dev/ansible.cfg) = /home/mator/ansible.dev/log
DEVEL_WARNING(/home/mator/ansible.dev/ansible.cfg) = False
INTERPRETER_PYTHON(/home/mator/ansible.dev/ansible.cfg) = auto_silent

[mator@archlinux ansible.dev]$ ansible --version
ansible 2.11.0.dev0 (devel b6360dc5e0) last updated 2020/10/23 18:50:01 (GMT +300)
  config file = /home/mator/ansible.dev/ansible.cfg
  configured module search path = ['/home/mator/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/mator/ansible.dev/ansible/lib/ansible
  ansible collection location = /home/mator/ansible.dev/collections
  executable location = /home/mator/ansible.dev/ansible/bin/ansible
  python version = 3.8.6 (default, Sep 30 2020, 04:00:38) [GCC 10.2.0]
  libyaml = True

[mator@archlinux ansible.dev]$ ansible -m setup localhost -a 'gather_subset=min'
localhost | SUCCESS => {
    "ansible_facts": {
...
```
^^ full output of `setup` module output follows
Looking at the code, I'm surprised if it really works: https://github.com/ansible-collections/community.general/blob/main/plugins/cache/redis.py#L128
(Or maybe the redis Python library got some magic to extract the correct IPv6 address, but it was only added later?)
> i can't reproduce issue:
> 
> ```
> [mator@archlinux ansible.dev]$ cat ansible.cfg 
> [defaults]
> inventory = hosts
> # gather_timeout = 20
> # debug=true
> log_path = ./log
> ANSIBLE_NOCOWS = 1
> interpreter_python = auto_silent
> collections_paths = ./collections
> pipelining = ANSIBLE_PIPELINING
> DEVEL_WARNING = 0
> fact_caching_connection = [::1]
> 

Can you reproduce it with your setup if you explicitly specify a port number? Because that was also done in the reported scenario.
@branleb can you please tell how do i specify it ? i'm actually no idea what is `CACHE_PLUGIN_CONNECTION`
@branleb Can you please provide the required details? Thanks.

needs_info
> @branleb can you please tell how do i specify it ? i'm actually no idea what is `CACHE_PLUGIN_CONNECTION`

e.g.:
`fact_caching_connection = "[::1]:6379:0"`
which produces the initially mentioned error - whereas 
`fact_caching_connection = "localhost:6379:0"`
or
`fact_caching_connection = "127.0.0.1:6379:0"`
work fine.

I just tried the case you specified as cannot reproduce (bracketed IPv6 localhost, no quoting) 
`fact_caching_connection = [::1]`
(and, just to be sure I tried 
`fact_caching_connection = [::1]:6379:0`
too)

Didn't work for me either. I got the same result as initially described:

`PLAY [ansible] ****************************************************************************************************************************************************************************************************************************************************************
ERROR! Unexpected Exception, this is probably a bug: invalid literal for int() with base 10: ''
the full traceback was:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 988, in get_connection
    connection = self._available_connections.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 123, in <module>
    exit_code = cli.run()
  File "/usr/lib/python3/dist-packages/ansible/cli/playbook.py", line 127, in run
    results = pbex.run()
  File "/usr/lib/python3/dist-packages/ansible/executor/playbook_executor.py", line 169, in run
    result = self._tqm.run(play=play)
  File "/usr/lib/python3/dist-packages/ansible/executor/task_queue_manager.py", line 241, in run
    play_return = strategy.run(iterator, play_context)
  File "/usr/lib/python3/dist-packages/ansible/plugins/strategy/linear.py", line 282, in run
    _hosts=self._hosts_cache, _hosts_all=self._hosts_cache_all)
  File "/usr/lib/python3/dist-packages/ansible/vars/manager.py", line 305, in get_vars
    facts = wrap_var(self._fact_cache.get(host.name, {}))
  File "/usr/lib/python3.7/_collections_abc.py", line 660, in get
    return self[key]
  File "/usr/lib/python3/dist-packages/ansible/vars/fact_cache.py", line 31, in __getitem__
    if not self._plugin.contains(key):
  File "/usr/lib/python3/dist-packages/ansible/plugins/cache/redis.py", line 132, in contains
    self._expire_keys()
  File "/usr/lib/python3/dist-packages/ansible/plugins/cache/redis.py", line 125, in _expire_keys
    self._db.zremrangebyscore(self._keys_set, 0, expiry_age)
  File "/usr/lib/python3/dist-packages/redis/client.py", line 2532, in zremrangebyscore
    return self.execute_command('ZREMRANGEBYSCORE', name, min, max)
  File "/usr/lib/python3/dist-packages/redis/client.py", line 772, in execute_command
    connection = pool.get_connection(command_name, **options)
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 990, in get_connection
    connection = self.make_connection()
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 1025, in make_connection
    return self.connection_class(**self.connection_kwargs)
  File "/usr/lib/python3/dist-packages/redis/connection.py", line 452, in __init__
    self.port = int(port)
ValueError: invalid literal for int() with base 10: ''
`

So apparently, the bug has been fixed somewhere in beetween v2.9.6 and your v2.11.0.dev0?
@branleb Could you please test if your problem has been solved with latest ansible-core and ansible releases?

needs_info
@branleb I took a look at the code, and the fact that it was using `.split(':')` to sort those fields raised a red-flag in my mind. I have raised this PR above to replace that with a regular expression, that should be more solid.

In the long run, I believe we should deprecate the current syntax and try to use a more standard url, maybe in the like of:
```
redis://[::1]:6379/0
```
But that should be a different PR than this one.

Resolving patch:
diff --git a/plugins/cache/redis.py b/plugins/cache/redis.py
--- a/plugins/cache/redis.py
+++ b/plugins/cache/redis.py
@@ -61,6 +61,7 @@
         type: integer
 '''
 
+import re
 import time
 import json
 
@@ -91,6 +92,8 @@ class CacheModule(BaseCacheModule):
     performance.
     """
     _sentinel_service_name = None
+    re_url_conn = re.compile(r'^([^:]+|\[[^]]+\]):(\d+):(\d+)(?::(.*))?$')
+    re_sent_conn = re.compile(r'^(.*):(\d+)$')
 
     def __init__(self, *args, **kwargs):
         uri = ''
@@ -130,11 +133,18 @@ def __init__(self, *args, **kwargs):
             self._db = self._get_sentinel_connection(uri, kw)
         # normal connection
         else:
-            connection = uri.split(':')
+            connection = self._parse_connection(self.re_url_conn, uri)
             self._db = StrictRedis(*connection, **kw)
 
         display.vv('Redis connection: %s' % self._db)
 
+    @staticmethod
+    def _parse_connection(re_patt, uri):
+        match = re_patt.match(uri)
+        if not match:
+            raise AnsibleError("Unable to parse connection string")
+        return match.groups()
+
     def _get_sentinel_connection(self, uri, kw):
         """
         get sentinel connection details from _uri
@@ -158,7 +168,7 @@ def _get_sentinel_connection(self, uri, kw):
             except IndexError:
                 pass  # password is optional
 
-        sentinels = [tuple(shost.split(':')) for shost in connections]
+        sentinels = [self._parse_connection(self.re_sent_conn, shost) for shost in connections]
         display.vv('\nUsing redis sentinels: %s' % sentinels)
         scon = Sentinel(sentinels, **kw)
         try:


Test patch:
diff --git a/tests/unit/plugins/cache/test_redis.py b/tests/unit/plugins/cache/test_redis.py
--- a/tests/unit/plugins/cache/test_redis.py
+++ b/tests/unit/plugins/cache/test_redis.py
@@ -23,10 +23,23 @@
 
 pytest.importorskip('redis')
 
+from ansible import constants as C
 from ansible.plugins.loader import cache_loader
+from ansible.release import __version__ as ansible_version
 from ansible_collections.community.general.plugins.cache.redis import CacheModule as RedisCache
 
 
 def test_redis_cachemodule():
     # The _uri option is required for the redis plugin
-    assert isinstance(cache_loader.get('community.general.redis', **{'_uri': '127.0.0.1:6379:1'}), RedisCache)
+    connection = '127.0.0.1:6379:1'
+    if ansible_version.startswith('2.9.'):
+        C.CACHE_PLUGIN_CONNECTION = connection
+    assert isinstance(cache_loader.get('community.general.redis', **{'_uri': connection}), RedisCache)
+
+
+def test_redis_cachemodule():
+    # The _uri option is required for the redis plugin
+    connection = '[::1]:6379:1'
+    if ansible_version.startswith('2.9.'):
+        C.CACHE_PLUGIN_CONNECTION = connection
+    assert isinstance(cache_loader.get('community.general.redis', **{'_uri': connection}), RedisCache)