Repository: ansible/ansible

Issue:
Redis fact caching fails to handle cache expirations mid-run
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
redis.py

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible --version
ansible 2.3.0.0
  config file = /home/dkimsey/redis/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Nov 20 2015, 02:00:19) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
```

##### CONFIGURATION
```
[defaults]
gathering = smart

# just demo a small amount please
gather_subset = virtual

fact_caching = redis
fact_caching_connection = localhost:6379:1
# 5s
fact_caching_timeout = 5

[privilege_escalation]
become=False
```

##### OS / ENVIRONMENT
RHEL 7/OS X

##### SUMMARY
The redis cache plugin will not return facts when the data expires mid-run. It raises a KeyError but that does not seem to trigger a re-gather-facts for those hosts. 

The jsonfile and in-memory facts appear to return the old, expired data.

I believe the desired behavior should be a KeyError in `smart` mode should trigger a re-gathering of facts for those hosts.

##### STEPS TO REPRODUCE

Using the aforementioned ansible.cfg settings, the following playbook will demo the expiring between plays. Note if one were to use jsonfile or no-cache (in-memory?) it works correctly and maintains the data the entire run.

**Playbook: `demo.yml`:**
```yaml

- hosts: localhost
  connection: local
  tasks:
      - name: Facts gathereed OK
        debug:
            var: ansible_date_time.epoch
      - name: Sleep for a few seconds
        command: sleep 2

- hosts: localhost
  connection: local
  tasks:
      - name: Facts from previous run OK
        debug:
            var: ansible_date_time.epoch
      - name: Sleep for 4s
        command: sleep 4

- hosts: localhost
  connection: local
  tasks:
      - name: Facts from previous run NOT OK
        debug:
            var: ansible_date_time.epoch
```

**Command:**
```
ansible-playbook -i localhost, ./demo.yml
```

##### EXPECTED RESULTS

Either maintain consistency with other fact caching mechanisms and the fact should never change during the run.
OR
A more-correct solution I feel is the third Play should refresh expired facts.

##### ACTUAL RESULTS

<!--- Paste verbatim command output between quotes below -->
```
$ ansible-playbook -i localhost, ./demo.yml

PLAY [localhost] *******************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************
ok: [localhost]

TASK [Facts gathereed OK] **********************************************************************************************
ok: [localhost] => {
    "ansible_date_time.epoch": "1504289061",
    "changed": false
}

TASK [Sleep for a few seconds] *****************************************************************************************
changed: [localhost]

PLAY [localhost] *******************************************************************************************************

TASK [Facts from previous run OK] **************************************************************************************
ok: [localhost] => {
    "ansible_date_time.epoch": "1504289061",
    "changed": false
}

TASK [Sleep for 4s] ****************************************************************************************************
changed: [localhost]

PLAY [localhost] *******************************************************************************************************

TASK [Facts from previous run NOT OK] **********************************************************************************
ok: [localhost] => {
    "ansible_date_time.epoch": "VARIABLE IS NOT DEFINED!",
    "changed": false
}

PLAY RECAP *************************************************************************************************************
localhost                  : ok=6    changed=2    unreachable=0    failed=0

```



Discussion:
cc @slok
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
cc @slok
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I've been poking at this a bit. I don't really grasp how gather_facts is being invoked and when the facts are cleared. The magic gather_facts task seems to be injected by PlayIterator's `__init__` but I'm having severe trouble following its execution flow. I was hoping to add logic to test for expirations/KeyErrors from the fact cache and if so, re-schedule a gather facts.

Regardless, as far as I can tell the redis.py cache module's methods are invoked only at the beginning of the playbook and never again. Which suggests the expiry is happening elsewhere, however I cannot figure out where.

Resolving patch:
diff --git a/lib/ansible/plugins/cache/__init__.py b/lib/ansible/plugins/cache/__init__.py
--- a/lib/ansible/plugins/cache/__init__.py
+++ b/lib/ansible/plugins/cache/__init__.py
@@ -105,29 +105,29 @@ def get(self, key):
         and it would be problematic if the key did expire after some long running tasks and
         user gets 'undefined' error in the same play """
 
-        if key in self._cache:
-            return self._cache.get(key)
+        if key not in self._cache:
 
-        if self.has_expired(key) or key == "":
-            raise KeyError
+            if self.has_expired(key) or key == "":
+                raise KeyError
 
-        cachefile = "%s/%s" % (self._cache_dir, key)
-        try:
+            cachefile = "%s/%s" % (self._cache_dir, key)
             try:
-                value = self._load(cachefile)
-                self._cache[key] = value
-                return value
-            except ValueError as e:
-                display.warning("error in '%s' cache plugin while trying to read %s : %s. "
-                                "Most likely a corrupt file, so erasing and failing." % (self.plugin_name, cachefile, to_bytes(e)))
-                self.delete(key)
-                raise AnsibleError("The cache file %s was corrupt, or did not otherwise contain valid data. "
-                                   "It has been removed, so you can re-run your command now." % cachefile)
-        except (OSError, IOError) as e:
-            display.warning("error in '%s' cache plugin while trying to read %s : %s" % (self.plugin_name, cachefile, to_bytes(e)))
-            raise KeyError
-        except Exception as e:
-            raise AnsibleError("Error while decoding the cache file %s: %s" % (cachefile, to_bytes(e)))
+                try:
+                    value = self._load(cachefile)
+                    self._cache[key] = value
+                except ValueError as e:
+                    display.warning("error in '%s' cache plugin while trying to read %s : %s. "
+                                    "Most likely a corrupt file, so erasing and failing." % (self.plugin_name, cachefile, to_bytes(e)))
+                    self.delete(key)
+                    raise AnsibleError("The cache file %s was corrupt, or did not otherwise contain valid data. "
+                                       "It has been removed, so you can re-run your command now." % cachefile)
+            except (OSError, IOError) as e:
+                display.warning("error in '%s' cache plugin while trying to read %s : %s" % (self.plugin_name, cachefile, to_bytes(e)))
+                raise KeyError
+            except Exception as e:
+                raise AnsibleError("Error while decoding the cache file %s: %s" % (cachefile, to_bytes(e)))
+
+        return self._cache.get(key)
 
     def set(self, key, value):
 
diff --git a/lib/ansible/plugins/cache/memcached.py b/lib/ansible/plugins/cache/memcached.py
--- a/lib/ansible/plugins/cache/memcached.py
+++ b/lib/ansible/plugins/cache/memcached.py
@@ -147,8 +147,9 @@ def __init__(self, *args, **kwargs):
 
         self._timeout = C.CACHE_PLUGIN_TIMEOUT
         self._prefix = C.CACHE_PLUGIN_PREFIX
-        self._cache = ProxyClientPool(connection, debug=0)
-        self._keys = CacheModuleKeys(self._cache, self._cache.get(CacheModuleKeys.PREFIX) or [])
+        self._cache = {}
+        self._db = ProxyClientPool(connection, debug=0)
+        self._keys = CacheModuleKeys(self._db, self._db.get(CacheModuleKeys.PREFIX) or [])
 
     def _make_key(self, key):
         return "{0}{1}".format(self._prefix, key)
@@ -159,17 +160,21 @@ def _expire_keys(self):
             self._keys.remove_by_timerange(0, expiry_age)
 
     def get(self, key):
-        value = self._cache.get(self._make_key(key))
-        # guard against the key not being removed from the keyset;
-        # this could happen in cases where the timeout value is changed
-        # between invocations
-        if value is None:
-            self.delete(key)
-            raise KeyError
-        return value
+        if key not in self._cache:
+            value = self._db.get(self._make_key(key))
+            # guard against the key not being removed from the keyset;
+            # this could happen in cases where the timeout value is changed
+            # between invocations
+            if value is None:
+                self.delete(key)
+                raise KeyError
+            self._cache[key] = value
+
+        return self._cache.get(key)
 
     def set(self, key, value):
-        self._cache.set(self._make_key(key), value, time=self._timeout, min_compress_len=1)
+        self._db.set(self._make_key(key), value, time=self._timeout, min_compress_len=1)
+        self._cache[key] = value
         self._keys.add(key)
 
     def keys(self):
@@ -181,7 +186,8 @@ def contains(self, key):
         return key in self._keys
 
     def delete(self, key):
-        self._cache.delete(self._make_key(key))
+        del self._cache[key]
+        self._db.delete(self._make_key(key))
         self._keys.discard(key)
 
     def flush(self):
diff --git a/lib/ansible/plugins/cache/redis.py b/lib/ansible/plugins/cache/redis.py
--- a/lib/ansible/plugins/cache/redis.py
+++ b/lib/ansible/plugins/cache/redis.py
@@ -17,7 +17,6 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-import sys
 import time
 import json
 
@@ -48,47 +47,55 @@ def __init__(self, *args, **kwargs):
 
         self._timeout = float(C.CACHE_PLUGIN_TIMEOUT)
         self._prefix = C.CACHE_PLUGIN_PREFIX
-        self._cache = StrictRedis(*connection)
+        self._cache = {}
+        self._db = StrictRedis(*connection)
         self._keys_set = 'ansible_cache_keys'
 
     def _make_key(self, key):
         return self._prefix + key
 
     def get(self, key):
-        value = self._cache.get(self._make_key(key))
-        # guard against the key not being removed from the zset;
-        # this could happen in cases where the timeout value is changed
-        # between invocations
-        if value is None:
-            self.delete(key)
-            raise KeyError
-        return json.loads(value)
+
+        if key not in self._cache:
+            value = self._db.get(self._make_key(key))
+            # guard against the key not being removed from the zset;
+            # this could happen in cases where the timeout value is changed
+            # between invocations
+            if value is None:
+                self.delete(key)
+                raise KeyError
+            self._cache[key] = json.loads(value)
+
+        return self._cache.get(key)
 
     def set(self, key, value):
+
         value2 = json.dumps(value)
         if self._timeout > 0:  # a timeout of 0 is handled as meaning 'never expire'
-            self._cache.setex(self._make_key(key), int(self._timeout), value2)
+            self._db.setex(self._make_key(key), int(self._timeout), value2)
         else:
-            self._cache.set(self._make_key(key), value2)
+            self._db.set(self._make_key(key), value2)
 
-        self._cache.zadd(self._keys_set, time.time(), key)
+        self._db.zadd(self._keys_set, time.time(), key)
+        self._cache[key] = value
 
     def _expire_keys(self):
         if self._timeout > 0:
             expiry_age = time.time() - self._timeout
-            self._cache.zremrangebyscore(self._keys_set, 0, expiry_age)
+            self._db.zremrangebyscore(self._keys_set, 0, expiry_age)
 
     def keys(self):
         self._expire_keys()
-        return self._cache.zrange(self._keys_set, 0, -1)
+        return self._db.zrange(self._keys_set, 0, -1)
 
     def contains(self, key):
         self._expire_keys()
-        return (self._cache.zrank(self._keys_set, key) is not None)
+        return (self._db.zrank(self._keys_set, key) is not None)
 
     def delete(self, key):
-        self._cache.delete(self._make_key(key))
-        self._cache.zrem(self._keys_set, key)
+        del self.cache[key]
+        self._db.delete(self._make_key(key))
+        self._db.zrem(self._keys_set, key)
 
     def flush(self):
         for key in self.keys():


Test patch: