Repository: opsdroid/opsdroid

Issue:
Redis database keeps auto-reconnecting
<!-- Before you post an issue or if you are unsure about something join our gitter channel https://gitter.im/opsdroid/ and ask away! We are more than happy to help you. -->
# Description
I have found an issue with the Redis database, if you set Redis on your `config.yaml` but the server is not running, the client will try to reconnect to the server ad infinitum.

This is possibly an issue with `asyncio_redis` and I have raised an issue within the project ([#126](https://github.com/jonathanslenders/asyncio-redis/issues/126)).

I have tried to set `auto_reconnect` to False directly on the `connect` method, but had no change.

The`asyncio_redis` source code uses an infinite loop when trying to reconnect and this is triggered whenever the client fails to connect to the server, it should be turned off if `auto_reconnect` is set to `False` but it doesn't seem to be happening.

Finally, when we try to terminal opsdroid with SIGINT the client is unable to the terminal due to the following exception:

```python
INFO opsdroid.core: Stopping database ...
ERROR opsdroid.core: Caught exception
ERROR opsdroid.core: {'message': 'Task exception was never retrieved', 
'exception': AttributeError("'NoneType' object has no attribute 'close'",), 
'future': <Task finished coro=<OpsDroid.handle_signal() done, defined at /Users/fabiorosado/Documents/GitHub/opsdroid/opsdroid/core.py:120> exception=AttributeError("'NoneType' object has no attribute 'close'",)>}
```

And the asyncio_redis will keep that infinite loop until you close the terminal window 😬 

Since this is an issue with `asyncio_redis` there isn't much we can do at the moment, the only thing we could try to do would be fix the `AttributeError` on `opsdroid.core.unload` - perhaps pass if this exception is raised?

## Steps to Reproduce

* Add Redis database to your `config.yaml`
* run opsdroid without starting the Redis database


## Expected Functionality
asyncio_redis shouldn't keep trying to reconnect to the client unless `reconnect` was set to true. This flag doesn't seem to be working as expected.


## Experienced Functionality
Explain what happened instead(Please include the debug log).

## Versions
- **Opsdroid version:** dev
- **Python version:** 3.7
- **OS/Docker version:** macOS 10.14.2

## Configuration File
Please include your version of the configuration file bellow.

```yaml
databases:
   - name: redis

```



<!-- Love opsdroid? Please consider supporting our collective:
 +👉  https://opencollective.com/opsdroid/donate -->


Discussion:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Resolving patch:
diff --git a/opsdroid/database/redis/__init__.py b/opsdroid/database/redis/__init__.py
--- a/opsdroid/database/redis/__init__.py
+++ b/opsdroid/database/redis/__init__.py
@@ -1,11 +1,14 @@
 """Module for storing data within Redis."""
-from datetime import date, datetime
 import json
-import time
+import logging
 
-import asyncio_redis
+import aioredis
+from aioredis import parser
 
 from opsdroid.database import Database
+from opsdroid.helper import JSONEncoder, JSONDecoder
+
+_LOGGER = logging.getLogger(__name__)
 
 
 class RedisDatabase(Database):
@@ -30,7 +33,7 @@ def __init__(self, config, opsdroid=None):
         self.port = self.config.get("port", 6379)
         self.database = self.config.get("database", 0)
         self.password = self.config.get("password", None)
-        self.reconnect = self.config.get("reconnect", False)
+        _LOGGER.debug(_("Loaded redis database connector."))
 
     async def connect(self):
         """Connect to the database.
@@ -39,13 +42,26 @@ async def connect(self):
         connect to Redis on localhost on port 6379
 
         """
-        self.client = await asyncio_redis.Connection.create(
-            host=self.host,
-            port=self.port,
-            db=self.database,
-            auto_reconnect=self.reconnect,
-            password=self.password,
-        )
+        try:
+            self.client = await aioredis.create_pool(
+                address=(self.host, int(self.port)),
+                db=self.database,
+                password=self.password,
+                parser=parser.PyReader,
+            )
+
+            _LOGGER.info(
+                _("Connected to redis database %s from %s on port %s"),
+                self.database,
+                self.host,
+                self.port,
+            )
+        except OSError:
+            _LOGGER.warning(
+                _("Unable to connect to redis database on address: %s port: %s"),
+                self.host,
+                self.port,
+            )
 
     async def put(self, key, data):
         """Store the data object in Redis against the key.
@@ -55,8 +71,9 @@ async def put(self, key, data):
             data (object): The data object to store.
 
         """
-        data = self.convert_object_to_timestamp(data)
-        await self.client.set(key, json.dumps(data))
+        if self.client:
+            _LOGGER.debug(_("Putting %s into redis"), key)
+            await self.client.execute("SET", key, json.dumps(data, cls=JSONEncoder))
 
     async def get(self, key):
         """Get data from Redis for a given key.
@@ -69,56 +86,16 @@ async def get(self, key):
                             object found for that key.
 
         """
-        data = await self.client.get(key)
+        if self.client:
+            _LOGGER.debug(_("Getting %s from redis"), key)
+            data = await self.client.execute("GET", key)
 
-        if data:
-            return self.convert_timestamp_to_object(json.loads(data))
+            if data:
+                return json.loads(data, encoding=JSONDecoder)
 
-        return None
+            return None
 
     async def disconnect(self):
         """Disconnect from the database."""
-        self.client.close()
-
-    @staticmethod
-    def convert_object_to_timestamp(data):
-        """
-        Serialize dict before storing into Redis.
-
-        Args:
-            dict: Dict to serialize
-
-        Returns:
-            dict: Dict from redis to unserialize
-
-        """
-        for k, value in data.items():
-            if isinstance(value, (datetime, date)):
-                value = "::".join(
-                    [type(value).__name__, "%d" % time.mktime(value.timetuple())]
-                )
-                data[k] = value
-        return data
-
-    @staticmethod
-    def convert_timestamp_to_object(data):
-        """
-        Unserialize data from Redis.
-
-        Args:
-            dict: Dict from redis to unserialize
-
-        Returns:
-            dict: Dict to serialize
-
-        """
-        for k, value in data.items():
-            value_type = value.split("::", 1)[0]
-            if value_type == "datetime":
-                timestamp = int(value.split("::", 1)[1])
-                value = datetime.fromtimestamp(timestamp)
-            elif value_type == "date":
-                timestamp = int(value.split("::", 1)[1])
-                value = date.fromtimestamp(timestamp)
-            data[k] = value
-        return data
+        if self.client:
+            self.client.close()
diff --git a/opsdroid/database/sqlite/__init__.py b/opsdroid/database/sqlite/__init__.py
--- a/opsdroid/database/sqlite/__init__.py
+++ b/opsdroid/database/sqlite/__init__.py
@@ -2,11 +2,11 @@
 import os
 import logging
 import json
-import datetime
 import aiosqlite
 
 from opsdroid.const import DEFAULT_ROOT_PATH
 from opsdroid.database import Database
+from opsdroid.helper import JSONEncoder, JSONDecoder
 
 _LOGGER = logging.getLogger(__name__)
 
@@ -116,124 +116,3 @@ async def get(self, key):
 
         self.client = _db
         return data
-
-
-class JSONEncoder(json.JSONEncoder):
-    """A extended JSONEncoder class.
-
-    This class is customised JSONEncoder class which helps to convert
-    dict to JSON. The datetime objects are converted to dict with fields
-    as keys.
-
-    """
-
-    # pylint: disable=method-hidden
-    # See https://github.com/PyCQA/pylint/issues/414 for reference
-
-    serializers = {}
-
-    def default(self, o):
-        """Convert the given datetime object to dict.
-
-        Args:
-            o (object): The datetime object to be marshalled.
-
-        Returns:
-            dict (object): A dict with datatime object data.
-
-        Example:
-            A dict which is returned after marshalling::
-
-                {
-                    "__class__": "datetime",
-                    "year": 2018,
-                    "month": 10,
-                    "day": 2,
-                    "hour": 0,
-                    "minute": 41,
-                    "second": 17,
-                    "microsecond": 74644
-                }
-
-        """
-        marshaller = self.serializers.get(type(o), super(JSONEncoder, self).default)
-        return marshaller(o)
-
-
-class JSONDecoder:
-    """A JSONDecoder class.
-
-    This class will convert dict containing datetime values
-    to datetime objects.
-
-    """
-
-    decoders = {}
-
-    def __call__(self, dct):
-        """Convert given dict to datetime objects.
-
-        Args:
-            dct (object): A dict containing datetime values and class type.
-
-        Returns:
-            object or dct: The datetime object for given dct, or dct if
-                             respective class decoder is not found.
-
-        Example:
-            A datetime object returned after decoding::
-
-                datetime.datetime(2018, 10, 2, 0, 41, 17, 74644)
-
-        """
-        if dct.get("__class__") in self.decoders:
-            return self.decoders[dct["__class__"]](dct)
-        return dct
-
-
-def register_json_type(type_cls, fields, decode_fn):
-    """Register JSON types.
-
-    This method will register the serializers and decoders for the
-    JSONEncoder and JSONDecoder classes respectively.
-
-    Args:
-        type_cls (object): A datetime object.
-        fields (list): List of fields used to store data in dict.
-        decode_fn (object): A lambda function object for decoding.
-
-    """
-    type_name = type_cls.__name__
-    JSONEncoder.serializers[type_cls] = lambda obj: dict(
-        __class__=type_name, **{field: getattr(obj, field) for field in fields}
-    )
-    JSONDecoder.decoders[type_name] = decode_fn
-
-
-register_json_type(
-    datetime.datetime,
-    ["year", "month", "day", "hour", "minute", "second", "microsecond"],
-    lambda dct: datetime.datetime(
-        dct["year"],
-        dct["month"],
-        dct["day"],
-        dct["hour"],
-        dct["minute"],
-        dct["second"],
-        dct["microsecond"],
-    ),
-)
-
-register_json_type(
-    datetime.date,
-    ["year", "month", "day"],
-    lambda dct: datetime.date(dct["year"], dct["month"], dct["day"]),
-)
-
-register_json_type(
-    datetime.time,
-    ["hour", "minute", "second", "microsecond"],
-    lambda dct: datetime.time(
-        dct["hour"], dct["minute"], dct["second"], dct["microsecond"]
-    ),
-)
diff --git a/opsdroid/helper.py b/opsdroid/helper.py
--- a/opsdroid/helper.py
+++ b/opsdroid/helper.py
@@ -1,10 +1,12 @@
 """Helper functions to use within OpsDroid."""
 
+import datetime
 import os
 import stat
 import shutil
 import logging
 import filecmp
+import json
 
 import nbformat
 from nbconvert import PythonExporter
@@ -154,3 +156,124 @@ def add_skill_attributes(func):
     if not hasattr(func, "constraints"):
         func.constraints = []
     return func
+
+
+class JSONEncoder(json.JSONEncoder):
+    """A extended JSONEncoder class.
+
+    This class is customised JSONEncoder class which helps to convert
+    dict to JSON. The datetime objects are converted to dict with fields
+    as keys.
+
+    """
+
+    # pylint: disable=method-hidden
+    # See https://github.com/PyCQA/pylint/issues/414 for reference
+
+    serializers = {}
+
+    def default(self, o):
+        """Convert the given datetime object to dict.
+
+        Args:
+            o (object): The datetime object to be marshalled.
+
+        Returns:
+            dict (object): A dict with datatime object data.
+
+        Example:
+            A dict which is returned after marshalling::
+
+                {
+                    "__class__": "datetime",
+                    "year": 2018,
+                    "month": 10,
+                    "day": 2,
+                    "hour": 0,
+                    "minute": 41,
+                    "second": 17,
+                    "microsecond": 74644
+                }
+
+        """
+        marshaller = self.serializers.get(type(o), super(JSONEncoder, self).default)
+        return marshaller(o)
+
+
+class JSONDecoder:
+    """A JSONDecoder class.
+
+    This class will convert dict containing datetime values
+    to datetime objects.
+
+    """
+
+    decoders = {}
+
+    def __call__(self, dct):
+        """Convert given dict to datetime objects.
+
+        Args:
+            dct (object): A dict containing datetime values and class type.
+
+        Returns:
+            object or dct: The datetime object for given dct, or dct if
+                             respective class decoder is not found.
+
+        Example:
+            A datetime object returned after decoding::
+
+                datetime.datetime(2018, 10, 2, 0, 41, 17, 74644)
+
+        """
+        if dct.get("__class__") in self.decoders:
+            return self.decoders[dct["__class__"]](dct)
+        return dct
+
+
+def register_json_type(type_cls, fields, decode_fn):
+    """Register JSON types.
+
+    This method will register the serializers and decoders for the
+    JSONEncoder and JSONDecoder classes respectively.
+
+    Args:
+        type_cls (object): A datetime object.
+        fields (list): List of fields used to store data in dict.
+        decode_fn (object): A lambda function object for decoding.
+
+    """
+    type_name = type_cls.__name__
+    JSONEncoder.serializers[type_cls] = lambda obj: dict(
+        __class__=type_name, **{field: getattr(obj, field) for field in fields}
+    )
+    JSONDecoder.decoders[type_name] = decode_fn
+
+
+register_json_type(
+    datetime.datetime,
+    ["year", "month", "day", "hour", "minute", "second", "microsecond"],
+    lambda dct: datetime.datetime(
+        dct["year"],
+        dct["month"],
+        dct["day"],
+        dct["hour"],
+        dct["minute"],
+        dct["second"],
+        dct["microsecond"],
+    ),
+)
+
+register_json_type(
+    datetime.date,
+    ["year", "month", "day"],
+    lambda dct: datetime.date(dct["year"], dct["month"], dct["day"]),
+)
+
+register_json_type(
+    datetime.time,
+    ["hour", "minute", "second", "microsecond"],
+    lambda dct: datetime.time(
+        dct["hour"], dct["minute"], dct["second"], dct["microsecond"]
+    ),
+)


Test patch:
diff --git a/tests/test_database_redis.py b/tests/test_database_redis.py
--- a/tests/test_database_redis.py
+++ b/tests/test_database_redis.py
@@ -1,5 +1,5 @@
 import asyncio
-import datetime
+import aioredis
 import unittest
 
 import asynctest
@@ -35,36 +35,7 @@ def test_init(self):
         self.assertEqual("localhost", database.host)
         self.assertEqual(6379, database.port)
         self.assertEqual(None, database.password)
-
-    def test_other(self):
-        unserialized_data = {
-            "example_string": "test",
-            "example_datetime": datetime.datetime.utcfromtimestamp(1538389815),
-            "example_date": datetime.date.fromtimestamp(1538366400),
-        }
-
-        serialized_data = RedisDatabase.convert_object_to_timestamp(unserialized_data)
-
-        self.assertEqual(serialized_data["example_string"], "test")
-        # Typically I would do assertDictEqual on the result, but as datetime are parsed based on the
-        # timezone of the computer it makes the unittest fragile depending on the timezone of the user.
-        self.assertEqual(serialized_data["example_datetime"][0:10], "datetime::")
-        self.assertEqual(serialized_data["example_date"][0:6], "date::")
-
-    def test_convert_timestamp_to_object(self):
-        serialized_data = {
-            "example_date": "date::1538366400",
-            "example_datetime": "datetime::1538389815",
-            "example_string": "test",
-        }
-
-        unserialized_data = RedisDatabase.convert_timestamp_to_object(serialized_data)
-
-        self.assertEqual(unserialized_data["example_string"], "test")
-        # Typically I would do assertDictEqual on the result, but as datetime are parsed based on the
-        # timezone of the computer it makes the unittest fragile depending on the timezone of the user.
-        self.assertIsInstance(unserialized_data["example_datetime"], datetime.datetime)
-        self.assertIsInstance(unserialized_data["example_date"], datetime.date)
+        self.assertLogs("_LOGGER", "debug")
 
 
 class TestRedisDatabaseAsync(asynctest.TestCase):
@@ -73,21 +44,41 @@ class TestRedisDatabaseAsync(asynctest.TestCase):
     async def test_connect(self):
         opsdroid = amock.CoroutineMock()
         database = RedisDatabase({}, opsdroid=opsdroid)
-        import asyncio_redis
 
-        with amock.patch.object(
-            asyncio_redis.Connection, "create"
-        ) as mocked_connection:
+        with amock.patch.object(aioredis, "create_pool") as mocked_connection:
             mocked_connection.side_effect = NotImplementedError
 
             with suppress(NotImplementedError):
                 await database.connect()
                 self.assertTrue(mocked_connection.called)
+                self.assertLogs("_LOGGER", "info")
+
+    async def test_connect_failure(self):
+        opsdroid = amock.CoroutineMock()
+        database = RedisDatabase({}, opsdroid=opsdroid)
+
+        with amock.patch.object(aioredis, "create_pool") as mocked_connection:
+            mocked_connection.side_effect = OSError()
+
+            with suppress(OSError):
+                await database.connect()
+                self.assertLogs("_LOGGER", "warning")
+
+    async def test_connect_logging(self):
+        opsdroid = amock.CoroutineMock()
+        database = RedisDatabase({}, opsdroid=opsdroid)
+
+        with amock.patch.object(aioredis, "create_pool") as mocked_connection:
+            mocked_connection.set_result = amock.CoroutineMock()
+
+            await database.connect()
+            self.assertTrue(mocked_connection.called)
+            self.assertLogs("_LOGGER", "info")
 
     async def test_get(self):
         db = RedisDatabase({})
         db.client = MockRedisClient()
-        db.client.get = amock.CoroutineMock(return_value='{"key":"value"}')
+        db.client.execute = amock.CoroutineMock(return_value='{"key":"value"}')
 
         result = await db.get("string")
 
@@ -96,7 +87,7 @@ async def test_get(self):
     async def test_get_return_None(self):
         db = RedisDatabase({})
         db.client = MockRedisClient()
-        db.client.get = amock.CoroutineMock(return_value=None)
+        db.client.execute = amock.CoroutineMock(return_value=None)
 
         result = await db.get("string")
 
@@ -105,7 +96,7 @@ async def test_get_return_None(self):
     async def test_put(self):
         db = RedisDatabase({})
         db.client = MockRedisClient()
-        db.client.set = amock.CoroutineMock(return_value='{"key":"value"}')
+        db.client.execute = amock.CoroutineMock(return_value='{"key":"value"}')
 
         result = await db.put("string", dict(key="value"))
 
diff --git a/tests/test_database_sqlite.py b/tests/test_database_sqlite.py
--- a/tests/test_database_sqlite.py
+++ b/tests/test_database_sqlite.py
@@ -1,15 +1,11 @@
 """Tests for the DatabaseSqlite class."""
 import asyncio
-import json
-import datetime
 
 import unittest
-import unittest.mock as mock
 import asynctest
 import asynctest.mock as amock
 
-from opsdroid.database.sqlite import DatabaseSqlite, JSONEncoder, JSONDecoder
-from opsdroid.database.sqlite import register_json_type
+from opsdroid.database.sqlite import DatabaseSqlite
 from opsdroid.__main__ import configure_lang
 
 
@@ -88,71 +84,3 @@ async def test_get_and_put(self):
             self.assertEqual("opsdroid", database.table)
             self.assertEqual({}, data)
             self.assertEqual("Connection", type(database.client).__name__)
-
-
-class TestJSONEncoder(unittest.TestCase):
-    """A JSON Encoder test class.
-
-    Test the custom json encoder class.
-
-    """
-
-    def setUp(self):
-        self.loop = asyncio.new_event_loop()
-
-    def test_datetime_to_dict(self):
-        """Test default of json encoder class.
-
-        This method will test the conversion of the datetime
-        object to dict.
-
-        """
-        type_cls = datetime.datetime
-        test_obj = datetime.datetime(2018, 10, 2, 0, 41, 17, 74644)
-        encoder = JSONEncoder()
-        obj = encoder.default(o=test_obj)
-        self.assertEqual(
-            {
-                "__class__": type_cls.__name__,
-                "year": 2018,
-                "month": 10,
-                "day": 2,
-                "hour": 0,
-                "minute": 41,
-                "second": 17,
-                "microsecond": 74644,
-            },
-            obj,
-        )
-
-
-class TestJSONDecoder(unittest.TestCase):
-    """A JSON Decoder test class.
-
-    Test the custom json decoder class.
-
-    """
-
-    def setUp(self):
-        self.loop = asyncio.new_event_loop()
-
-    def test_dict_to_datetime(self):
-        """Test call of json decoder class.
-
-        This method will test the conversion of the dict to
-        datetime object.
-
-        """
-        test_obj = {
-            "__class__": datetime.datetime.__name__,
-            "year": 2018,
-            "month": 10,
-            "day": 2,
-            "hour": 0,
-            "minute": 41,
-            "second": 17,
-            "microsecond": 74644,
-        }
-        decoder = JSONDecoder()
-        obj = decoder(test_obj)
-        self.assertEqual(datetime.datetime(2018, 10, 2, 0, 41, 17, 74644), obj)
diff --git a/tests/test_helper.py b/tests/test_helper.py
--- a/tests/test_helper.py
+++ b/tests/test_helper.py
@@ -1,4 +1,6 @@
 import os
+import asyncio
+import datetime
 import tempfile
 import unittest
 import unittest.mock as mock
@@ -10,6 +12,8 @@
     convert_ipynb_to_script,
     extract_gist_id,
     get_opsdroid,
+    JSONEncoder,
+    JSONDecoder,
 )
 
 
@@ -67,3 +71,71 @@ def test_extract_gist_id(self):
     def test_opsdroid(self):
         # Test that get_opsdroid returns None if no instances exist
         assert get_opsdroid() is None
+
+
+class TestJSONEncoder(unittest.TestCase):
+    """A JSON Encoder test class.
+
+    Test the custom json encoder class.
+
+    """
+
+    def setUp(self):
+        self.loop = asyncio.new_event_loop()
+
+    def test_datetime_to_dict(self):
+        """Test default of json encoder class.
+
+        This method will test the conversion of the datetime
+        object to dict.
+
+        """
+        type_cls = datetime.datetime
+        test_obj = datetime.datetime(2018, 10, 2, 0, 41, 17, 74644)
+        encoder = JSONEncoder()
+        obj = encoder.default(o=test_obj)
+        self.assertEqual(
+            {
+                "__class__": type_cls.__name__,
+                "year": 2018,
+                "month": 10,
+                "day": 2,
+                "hour": 0,
+                "minute": 41,
+                "second": 17,
+                "microsecond": 74644,
+            },
+            obj,
+        )
+
+
+class TestJSONDecoder(unittest.TestCase):
+    """A JSON Decoder test class.
+
+    Test the custom json decoder class.
+
+    """
+
+    def setUp(self):
+        self.loop = asyncio.new_event_loop()
+
+    def test_dict_to_datetime(self):
+        """Test call of json decoder class.
+
+        This method will test the conversion of the dict to
+        datetime object.
+
+        """
+        test_obj = {
+            "__class__": datetime.datetime.__name__,
+            "year": 2018,
+            "month": 10,
+            "day": 2,
+            "hour": 0,
+            "minute": 41,
+            "second": 17,
+            "microsecond": 74644,
+        }
+        decoder = JSONDecoder()
+        obj = decoder(test_obj)
+        self.assertEqual(datetime.datetime(2018, 10, 2, 0, 41, 17, 74644), obj)