Repository: opsdroid/opsdroid

Issue:
Move Matrix connector into core
This issue covers adding the [Matrix connector](https://github.com/opsdroid/connector-matrix) to core.

## Background
A while ago we began moving connectors from external plugins into the core of the project (see #185 for more context). We started with [slack](https://github.com/opsdroid/opsdroid/blob/master/opsdroid/connector/slack/__init__.py) and [websockets](https://github.com/opsdroid/opsdroid/blob/master/opsdroid/connector/websocket/__init__.py) but need to go through all the other existing plugins and move them into the core.

## Steps

 - Make a new submodule directory in [`opsdroid.connector`](https://github.com/opsdroid/opsdroid/tree/master/opsdroid/connector) and copy the connector code over.
 - Update the [`requirements.txt`](https://github.com/opsdroid/opsdroid/blob/master/requirements.txt) with any dependencies from the connector if necessary.
 - Write tests for the connector. (See the [Slack connector tests](https://github.com/jacobtomlinson/opsdroid/blob/master/tests/test_connector_slack.py) for inspiration).
 - Copy the relevant information from the connector `README.md` into a [new documentation page](https://github.com/opsdroid/opsdroid/tree/master/docs/connectors).
 - Add the new page to the [mkdocs.yml](https://github.com/opsdroid/opsdroid/blob/master/mkdocs.yml).
 - Add to the [list of connectors](https://github.com/opsdroid/opsdroid/blob/master/docs/configuration-reference.md#connector-modules).
 - Add a deprecation notice to the old connector. (See [the slack connector](https://github.com/opsdroid/connector-slack))


Discussion:
I recommend that @SolarDrew or @Cadair take this if you have bandwidth. However if someone else wants to take it then please go ahead!
Can you override the core connector with a plugin connector? i.e. for development or similar?
My instinct is that if you specify a path in the config then that should override the core connector. I need to check that this is actually true though. 

You can always configure a connector with a different name (ie `matrixdev`) and set the path on that to be sure. 

Resolving patch:
diff --git a/opsdroid/connector/matrix/__init__.py b/opsdroid/connector/matrix/__init__.py
new file mode 100644
--- /dev/null
+++ b/opsdroid/connector/matrix/__init__.py
@@ -0,0 +1,3 @@
+"""Imports the matrix connector."""
+
+from .connector import ConnectorMatrix  # noqa: F401
diff --git a/opsdroid/connector/matrix/connector.py b/opsdroid/connector/matrix/connector.py
new file mode 100644
--- /dev/null
+++ b/opsdroid/connector/matrix/connector.py
@@ -0,0 +1,233 @@
+"""Connector for Matrix (https://matrix.org)."""
+
+import re
+import logging
+
+import aiohttp
+
+from matrix_api_async.api_asyncio import AsyncHTTPAPI
+from matrix_client.errors import MatrixRequestError
+
+from opsdroid.connector import Connector
+from opsdroid.message import Message
+
+from .html_cleaner import clean
+
+
+_LOGGER = logging.getLogger(__name__)
+
+__all__ = ['ConnectorMatrix']
+
+
+class ConnectorMatrix(Connector):
+    """Connector for Matrix (https://matrix.org)."""
+
+    def __init__(self, config, opsdroid=None):  # noqa: D107
+        """Init the config for the connector."""
+        super().__init__(config, opsdroid=opsdroid)
+
+        self.name = "ConnectorMatrix"  # The name of your connector
+        self.config = config  # The config dictionary to be accessed later
+        self.rooms = config.get('rooms', None)
+        if not self.rooms:
+            self.rooms = {'main': config['room']}
+        self.room_ids = {}
+        self.default_room = self.rooms['main']
+        self.mxid = config['mxid']
+        self.nick = config.get('nick', None)
+        self.homeserver = config.get('homeserver', "https://matrix.org")
+        self.password = config['password']
+        self.room_specific_nicks = config.get("room_specific_nicks", False)
+        self.session = None
+        self.filter_id = None
+        self.connection = None
+
+    @property
+    def filter_json(self):
+        """Define JSON filter to apply to incoming events."""
+        return {
+            "event_format": "client",
+            "account_data": {
+                "limit": 0,
+                "types": []
+            },
+            "presence": {
+                "limit": 0,
+                "types": []
+            },
+            "room": {
+                "rooms": [],
+                "account_data": {
+                    "types": []
+                },
+                "timeline": {
+                    "types": ["m.room.message"]
+                },
+                "ephemeral": {
+                    "types": []
+                },
+                "state": {
+                    "types": []
+                }
+            }
+        }
+
+    async def make_filter(self, api, room_ids):
+        """Make a filter on the server for future syncs."""
+        fjson = self.filter_json
+        for room_id in room_ids:
+            fjson['room']['rooms'].append(room_id)
+
+        resp = await api.create_filter(
+            user_id=self.mxid, filter_params=fjson)
+
+        return resp['filter_id']
+
+    async def connect(self):
+        """Create connection object with chat library."""
+        session = aiohttp.ClientSession()
+        mapi = AsyncHTTPAPI(self.homeserver, session)
+
+        self.session = session
+        login_response = await mapi.login(
+            "m.login.password", user=self.mxid, password=self.password)
+        mapi.token = login_response['access_token']
+        mapi.sync_token = None
+
+        for roomname, room in self.rooms.items():
+            response = await mapi.join_room(room)
+            self.room_ids[roomname] = response['room_id']
+        self.connection = mapi
+
+        # Create a filter now, saves time on each later sync
+        self.filter_id = await self.make_filter(mapi, self.room_ids.values())
+
+        # Do initial sync so we don't get old messages later.
+        response = await self.connection.sync(
+            timeout_ms=3000,
+            filter='{ "room": { "timeline" : { "limit" : 1 } } }',
+            set_presence="online")
+        self.connection.sync_token = response["next_batch"]
+
+        if self.nick:
+            display_name = await self.connection.get_display_name(self.mxid)
+            if display_name != self.nick:
+                await self.connection.set_display_name(self.mxid, self.nick)
+
+    async def _parse_sync_response(self, response):
+        self.connection.sync_token = response["next_batch"]
+        for roomid in self.room_ids.values():
+            room = response['rooms']['join'].get(roomid, None)
+            if room and 'timeline' in room:
+                for event in room['timeline']['events']:
+                    if event['content']['msgtype'] == 'm.text':
+                        if event['sender'] != self.mxid:
+                            return Message(event['content']['body'],
+                                           await self._get_nick(
+                                               roomid,
+                                               event['sender']),
+                                           roomid, self,
+                                           raw_message=event)
+
+    async def listen(self):  # pragma: no cover
+        """Listen for new messages from the chat service."""
+        while True:  # pylint: disable=R1702
+            try:
+                response = await self.connection.sync(
+                    self.connection.sync_token,
+                    timeout_ms=int(6 * 60 * 60 * 1e3),  # 6h in ms
+                    filter=self.filter_id)
+                _LOGGER.debug("matrix sync request returned")
+                message = await self._parse_sync_response(response)
+                await self.opsdroid.parse(message)
+
+            except Exception:  # pylint: disable=W0703
+                _LOGGER.exception('Matrix Sync Error')
+
+    async def _get_nick(self, roomid, mxid):
+        """
+        Get nickname from user ID.
+
+        Get the nickname of a sender depending on the room specific config
+        setting.
+        """
+        if self.room_specific_nicks:
+            try:
+                return await self.connection.get_room_displayname(roomid, mxid)
+            except Exception:  # pylint: disable=W0703
+                # Fallback to the non-room specific one
+                logging.exception(
+                    "Failed to lookup room specific nick for %s", mxid)
+
+        try:
+            return await self.connection.get_display_name(mxid)
+        except MatrixRequestError as mre:
+            # Log the error if it's not the 404 from the user not having a nick
+            if mre.code != 404:
+                logging.exception("Failed to lookup nick for %s", mxid)
+            return mxid
+
+    @staticmethod
+    def _get_formatted_message_body(message, body=None, msgtype="m.text"):
+        """
+        Get HTML from a message.
+
+        Return the json representation of the message in
+        "org.matrix.custom.html" format.
+        """
+        # Markdown leaves a <p></p> around standard messages that we want to
+        # strip:
+        if message.startswith('<p>'):
+            message = message[3:]
+            if message.endswith('</p>'):
+                message = message[:-4]
+
+        clean_html = clean(message)
+
+        return {
+            # Strip out any tags from the markdown to make the body
+            "body": body if body else re.sub('<[^<]+?>', '', clean_html),
+            "msgtype": msgtype,
+            "format": "org.matrix.custom.html",
+            "formatted_body": clean_html
+            }
+
+    async def respond(self, message, room=None):
+        """Send `message.text` back to the chat service."""
+        if not room:
+            # Connector responds in the same room it received the original
+            # message
+            room_id = message.room
+        else:
+            room_id = self.rooms[room]
+
+        # Ensure we have a room id not alias
+        if not room_id.startswith('!'):
+            room_id = await self.connection.get_room_id(room_id)
+        else:
+            room_id = room_id
+
+        try:
+            await self.connection.send_message_event(
+                room_id,
+                "m.room.message",
+                self._get_formatted_message_body(message.text))
+        except aiohttp.client_exceptions.ServerDisconnectedError:
+            _LOGGER.debug("Server had disconnected, retrying send.")
+            await self.connection.send_message_event(
+                room_id,
+                "m.room.message",
+                self._get_formatted_message_body(message.text))
+
+    async def disconnect(self):
+        """Close the matrix session."""
+        await self.session.close()
+
+    def get_roomname(self, room):
+        """Get the name of a room from alias or room ID."""
+        if room.startswith(('#', '!')):
+            for connroom in self.rooms:
+                if room in (connroom, self.room_ids[connroom]):
+                    return connroom
+
+        return room
diff --git a/opsdroid/connector/matrix/html_cleaner.py b/opsdroid/connector/matrix/html_cleaner.py
new file mode 100644
--- /dev/null
+++ b/opsdroid/connector/matrix/html_cleaner.py
@@ -0,0 +1,76 @@
+"""Helpers for sanitising HTML input to the bot."""
+
+import bleach
+
+__all__ = ['clean']
+
+
+"""
+Take the list of allowed tags and attributes from Riot for consistency:
+https://github.com/matrix-org/matrix-react-sdk/blob/master/src/HtmlUtils.js#L180-L195
+"""
+
+ALLOWED_TAGS = [
+    'font',  # custom to matrix for IRC-style font coloring
+    'del',  # for markdown
+    'h1',
+    'h2',
+    'h3',
+    'h4',
+    'h5',
+    'h6',
+    'blockquote',
+    'p',
+    'a',
+    'ul',
+    'ol',
+    'sup',
+    'sub',
+    'nl',
+    'li',
+    'b',
+    'i',
+    'u',
+    'strong',
+    'em',
+    'strike',
+    'code',
+    'hr',
+    'br',
+    'div',
+    'table',
+    'thead',
+    'caption',
+    'tbody',
+    'tr',
+    'th',
+    'td',
+    'pre',
+    'span',
+    'img',
+]
+
+ALLOWED_ATTRIBUTES = {
+    'font': ['color', 'data-mx-bg-color', 'data-mx-color', 'style'],
+    'span': ['data-mx-bg-color', 'data-mx-color', 'style'],
+    'a': ['href', 'name', 'target', 'rel'],
+    'img': ['src', 'width', 'height', 'alt', 'title'],
+    'ol': ['start'],
+}
+
+
+def clean(html, **kwargs):
+    """
+    Sanitise HTML fragments.
+
+    A version of `bleach.clean` but with Riot's allowed tags and ``strip=True``
+    by default.
+    """
+    defaults = {
+        'strip': True,
+        'tags': ALLOWED_TAGS,
+        'attributes': ALLOWED_ATTRIBUTES
+    }
+    defaults.update(kwargs)
+
+    return bleach.clean(html, **defaults)


Test patch:
diff --git a/tests/test_connector_matrix.py b/tests/test_connector_matrix.py
new file mode 100644
--- /dev/null
+++ b/tests/test_connector_matrix.py
@@ -0,0 +1,290 @@
+"""Tests for the ConnectorMatrix class."""
+import asyncio
+from unittest import mock
+
+import aiohttp
+import asynctest
+import asynctest.mock as amock
+from matrix_api_async import AsyncHTTPAPI
+from matrix_client.errors import MatrixRequestError
+
+from opsdroid.core import OpsDroid
+from opsdroid.connector.matrix import ConnectorMatrix
+from opsdroid.__main__ import configure_lang  # noqa
+
+api_string = 'matrix_api_async.AsyncHTTPAPI.{}'
+
+
+def setup_connector():
+    """Initiate a basic connector setup for testing on"""
+    connector = ConnectorMatrix(
+        {"room": "#test:localhost",
+         "mxid": "@opsdroid:localhost",
+         "password": "hello",
+         "homeserver": "http://localhost:8008"}
+    )
+    return connector
+
+
+class TestConnectorMatrixAsync(asynctest.TestCase):
+    """Test the async methods of the opsdroid Matrix connector class."""
+    @property
+    def sync_return(self):
+        """Define some mock json to return from the sync method"""
+        return {
+            "account_data": {
+                "events": []
+            },
+            "device_lists": {
+                "changed": [],
+                "left": []
+            },
+            "device_one_time_keys_count": {
+                "signed_curve25519": 50
+            },
+            "groups": {
+                "invite": {},
+                "join": {},
+                "leave": {}
+            },
+            "next_batch": "s801873745",
+            "presence": {
+                "events": []
+            },
+            "rooms": {
+                "invite": {},
+                "join": {
+                    "!aroomid:localhost": {
+                        "account_data": {
+                            "events": []
+                        },
+                        "ephemeral": {
+                            "events": []
+                        },
+                        "state": {
+                            "events": []
+                        },
+                        "summary": {},
+                        "timeline": {
+                            "events": [
+                                {
+                                    "content": {
+                                        "body": "LOUD NOISES",
+                                        "msgtype": "m.text"
+                                    },
+                                    "event_id": "$eventid:localhost",
+                                    "origin_server_ts": 1547124373956,
+                                    "sender": "@cadair:cadair.com",
+                                    "type": "m.room.message",
+                                    "unsigned": {
+                                        "age": 3498
+                                    }
+                                }
+                            ],
+                            "limited": False,
+                            "prev_batch": "s801873709"
+                        },
+                        "unread_notifications": {
+                            "highlight_count": 0,
+                            "notification_count": 0
+                        }
+                    }
+                },
+                "leave": {}
+            },
+            "to_device": {
+                "events": []
+            }
+        }
+
+    def setUp(self):
+        """Basic setting up for tests"""
+        self.connector = setup_connector()
+        self.api = AsyncHTTPAPI('https://notaurl.com', None)
+        self.connector.connection = self.api
+
+    async def test_make_filter(self):
+        with amock.patch(api_string.format('create_filter')) as patched_filter:
+            patched_filter.return_value = asyncio.Future()
+            patched_filter.return_value.set_result({'filter_id': 'arbitrary string'})
+            test_rooms = ['!notaroom:matrix.org', '!notanotherroom:matrix.org']
+            filter_id = await self.connector.make_filter(self.api, test_rooms)
+            assert filter_id == 'arbitrary string'
+
+            assert patched_filter.called
+            assert patched_filter.call_args[1]['user_id'] == '@opsdroid:localhost'
+            assert patched_filter.call_args[1]['filter_params']['room']['rooms'] == test_rooms
+
+    async def test_connect(self):
+        with amock.patch(api_string.format('login')) as patched_login, \
+             amock.patch(api_string.format('join_room')) as patched_join_room, \
+             amock.patch(api_string.format('create_filter')) as patched_filter, \
+             amock.patch(api_string.format('sync')) as patched_sync, \
+             amock.patch(api_string.format('get_display_name')) as patched_get_nick, \
+             amock.patch(api_string.format('set_display_name')) as patch_set_nick, \
+             amock.patch('aiohttp.ClientSession') as patch_cs, \
+             OpsDroid() as opsdroid:
+
+            # Skip actually creating a client session
+            patch_cs.return_value = amock.MagicMock()
+
+            patched_login.return_value = asyncio.Future()
+            patched_login.return_value.set_result({'access_token': 'arbitrary string1'})
+
+            patched_join_room.return_value = asyncio.Future()
+            patched_join_room.return_value.set_result({'room_id': '!aroomid:localhost'})
+
+            patched_filter.return_value = asyncio.Future()
+            patched_filter.return_value.set_result({'filter_id': 'arbitrary string'})
+
+            patched_sync.return_value = asyncio.Future()
+            patched_sync.return_value.set_result({'next_batch': 'arbitrary string2'})
+
+            await self.connector.connect()
+
+            assert '!aroomid:localhost' in self.connector.room_ids.values()
+
+            assert self.connector.connection.token == 'arbitrary string1'
+
+            assert self.connector.filter_id == 'arbitrary string'
+
+            assert self.connector.connection.sync_token == 'arbitrary string2'
+
+            self.connector.nick = "Rabbit Hole"
+
+            patched_get_nick.return_value = asyncio.Future()
+            patched_get_nick.return_value.set_result("Rabbit Hole")
+
+            await self.connector.connect()
+
+            assert patched_get_nick.called
+            assert not patch_set_nick.called
+
+            patched_get_nick.return_value = asyncio.Future()
+            patched_get_nick.return_value.set_result("Neo")
+
+            self.connector.mxid = "@morpheus:matrix.org"
+
+            await self.connector.connect()
+
+            assert patched_get_nick.called
+            assert patch_set_nick.called_once_with("@morpheus:matrix.org", "Rabbit Hole")
+
+    async def test_parse_sync_response(self):
+        self.connector.room_ids = {'main': '!aroomid:localhost'}
+        self.connector.filter_id = 'arbitrary string'
+
+        with amock.patch(api_string.format('get_display_name')) as patched_name:
+            patched_name.return_value = asyncio.Future()
+            patched_name.return_value.set_result('SomeUsersName')
+
+            returned_message = await self.connector._parse_sync_response(self.sync_return)
+
+            assert returned_message.text == 'LOUD NOISES'
+            assert returned_message.user == 'SomeUsersName'
+            assert returned_message.room == '!aroomid:localhost'
+            assert returned_message.connector == self.connector
+            raw_message = self.sync_return['rooms']['join']['!aroomid:localhost']['timeline']['events'][0]
+            assert returned_message.raw_message == raw_message
+
+    async def test_get_nick(self):
+        self.connector.room_specific_nicks = True
+
+        with amock.patch(api_string.format('get_room_displayname')) as patched_roomname, \
+             amock.patch(api_string.format('get_display_name')) as patched_globname:
+            patched_roomname.return_value = asyncio.Future()
+            patched_roomname.return_value.set_result('')
+
+            mxid = '@notaperson:matrix.org'
+            assert await self.connector._get_nick('#notaroom:localhost', mxid) == ''
+            # Test if a room displayname couldn't be found
+            patched_roomname.side_effect = Exception()
+
+            # Test if that leads to a global displayname being returned
+            patched_globname.return_value = asyncio.Future()
+            patched_globname.return_value.set_result('@notaperson')
+            assert await self.connector._get_nick('#notaroom:localhost', mxid) == '@notaperson'
+
+            # Test that failed nickname lookup returns the mxid
+            patched_globname.side_effect = MatrixRequestError()
+            assert await self.connector._get_nick('#notaroom:localhost', mxid) == mxid
+
+    async def test_get_formatted_message_body(self):
+        original_html = "<p><h3><no>Hello World</no></h3></p>"
+        original_body = "### Hello World"
+        message = self.connector._get_formatted_message_body(original_html)
+        assert message['formatted_body'] == "<h3>Hello World</h3>"
+        assert message['body'] == "Hello World"
+
+        message = self.connector._get_formatted_message_body(original_html,
+                                                             original_body)
+        assert message['formatted_body'] == "<h3>Hello World</h3>"
+        assert message['body'] == "### Hello World"
+
+    async def _get_message(self):
+        self.connector.room_ids = {'main': '!aroomid:localhost'}
+        self.connector.filter_id = 'arbitrary string'
+        m = 'opsdroid.connector.matrix.ConnectorMatrix._get_nick'
+
+        with amock.patch(m) as patched_nick:
+            patched_nick.return_value = asyncio.Future()
+            patched_nick.return_value.set_result("Neo")
+
+            return await self.connector._parse_sync_response(self.sync_return)
+
+    async def test_respond_retry(self):
+        message = await self._get_message()
+        with amock.patch(api_string.format("send_message_event")) as patched_send:
+            patched_send.return_value = asyncio.Future()
+            patched_send.return_value.set_result(None)
+            await self.connector.respond(message)
+
+            message_obj = self.connector._get_formatted_message_body(message.text)
+            assert patched_send.called_once_with(message.room,
+                                                 "m.room.message",
+                                                 message_obj)
+
+            patched_send.side_effect = [aiohttp.client_exceptions.ServerDisconnectedError(),
+                                        patched_send.return_value]
+
+            await self.connector.respond(message)
+
+            message_obj = self.connector._get_formatted_message_body(message.text)
+            assert patched_send.called_once_with(message.room,
+                                                 "m.room.message",
+                                                 message_obj)
+
+    async def test_respond_room(self):
+        message = await self._get_message()
+        with amock.patch(api_string.format("send_message_event")) as patched_send, \
+             amock.patch(api_string.format("get_room_id")) as patched_room_id:
+
+            patched_send.return_value = asyncio.Future()
+            patched_send.return_value.set_result(None)
+
+            patched_room_id.return_value = asyncio.Future()
+            patched_room_id.return_value.set_result(message.room)
+
+            await self.connector.respond(message, room="main")
+
+            message_obj = self.connector._get_formatted_message_body(message.text)
+            assert patched_send.called_once_with(message.room,
+                                                 "m.room.message",
+                                                 message_obj)
+
+    async def test_disconnect(self):
+        self.connector.session = amock.MagicMock()
+        self.connector.session.close = amock.CoroutineMock()
+        await self.connector.disconnect()
+        assert self.connector.session.close.called
+
+    def test_get_roomname(self):
+        self.connector.rooms = ['#notthisroom:localhost',
+                                '#thisroom:localhost']
+        self.connector.room_ids = dict(zip(self.connector.rooms,
+                                           ['!aroomid:localhost',
+                                            '!anotherroomid:localhost']))
+
+        assert self.connector.get_roomname('#thisroom:localhost') == '#thisroom:localhost'
+        assert self.connector.get_roomname('!anotherroomid:localhost') == '#thisroom:localhost'
+        assert self.connector.get_roomname('someroom') == 'someroom'