Repository: easy-as-python/django-webmention

Issue:
response_body saving bytes instead of a decoded string
I noticed a subtle bug in the [resolution.py::fetch_and_validate_source](https://github.com/easy-as-python/django-webmention/blob/9faab3dad9529d14399eb8dad016370adbd3054a/src/webmention/resolution.py#L19-L27).

While response.content is a bytes-string ( [docs](https://requests.readthedocs.io/en/master/user/quickstart/#binary-response-content)), it's never decoded before saving to the database. As a result it saves as a string into the db as follows: `"b'\n<!DOCTYPE html><html><head><meta charset="utf-8" />\n<...."`. 

Practically speaking this is fine for pure ascii data, but any unicode characters aren't be properly decoded before saving, resulting in double-escaped characters and must be corrected before you can display it properly to the user. e.g. `"Çelik"` will be saved as `"\xc3\x87elik"`

```
>>> b'\xc3\x87elik'  # raw bytes from response.content
b'\xc3\x87elik'
>>> str(b'\xc3\x87elik')  # raw string conversion gets double esacped
"b'\\xc3\\x87elik'" 
>>> b'\xc3\x87elik'.decode("utf-8")  # properly decoded
'Çelik'
```

Existing response_bodies can be fixed using [ftfy](https://ftfy.readthedocs.io/en/latest/) package by running the string through something like this `ftfy.fix_text(ftfy.fixes.decode_escapes(mention.response_body))`. 

Changing fetch_and_validate_source to return `response.text`, which is simply the decoded response.content would fix this issue.  Looking for the target in `str(response.content)` might also have a subtle bug if you used a utf-8 domain as well, so I believe it should validate using response.text as well.  

Refactored I think it should look something like this:

```python
def fetch_and_validate_source(source, target):
    response = requests.get(source)
    if response.status_code == 200:
        if target in response.text:
            return response.text
        else:
            raise TargetNotFoundError("Source URL did not contain target URL")
    else:
        raise SourceFetchError("Could not fetch source URL")
```

Discussion:

Patch:
diff --git a/setup.cfg b/setup.cfg
index a6d40d4..65e0534 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -13,9 +13,6 @@ classifiers =
     Development Status :: 5 - Production/Stable
     Intended Audience :: Developers
     Framework :: Django
-    Framework :: Django :: 1.11
-    Framework :: Django :: 2.0
-    Framework :: Django :: 2.1
     Framework :: Django :: 2.2
     Framework :: Django :: 3.0
     Framework :: Django :: 3.1
@@ -32,7 +29,7 @@ classifiers =
 package_dir = =src
 packages = find:
 install_requires =
-    Django>=1.8.2
+    Django>=2.2.0
     requests>=2.7.0
 
 [options.packages.find]
@@ -72,15 +69,13 @@ python_files =
 addopts = -ra -q --cov=webmention
 
 [tox:tox]
-envlist = {py36,py37,py38}-django{1.11,2.1,2.2,3.0,3.1}
+envlist = {py36,py37,py38}-django{2.2,3.0,3.1}
 
 [testenv]
 extras = test
 commands =
     pytest {posargs}
 deps =
-    django1.11: Django>=1.11,<2
-    django2.1: Django>=2.1,<2.2
     django2.2: Django>=2.2,<2.3
     django3.0: Django>=3.0,<3.1
     django3.1: Django>=3.1,<3.2
diff --git a/src/webmention/resolution.py b/src/webmention/resolution.py
index 87bc418..d01f98e 100644
--- a/src/webmention/resolution.py
+++ b/src/webmention/resolution.py
@@ -19,8 +19,8 @@ def url_resolves(url):
 def fetch_and_validate_source(source, target):
     response = requests.get(source)
     if response.status_code == 200:
-        if target in str(response.content):
-            return response.content
+        if target in response.text:
+            return response.text
         else:
             raise TargetNotFoundError("Source URL did not contain target URL")
     else:


Test patch:
diff --git a/tests/test_resolution.py b/tests/test_resolution.py
index 501dcec..a404659 100644
--- a/tests/test_resolution.py
+++ b/tests/test_resolution.py
@@ -26,10 +26,10 @@ def test_url_resolves_when_does_not_resolve(mock_resolve):
 def test_fetch_and_validate_source_happy_path(mock_get, test_source, test_target):
     mock_response = Mock()
     mock_response.status_code = 200
-    mock_response.content = '<a href="{href}">{href}</a>'.format(href=test_target)
+    mock_response.text = '<a href="{href}">{href}</a>'.format(href=test_target)
     mock_get.return_value = mock_response
 
-    assert fetch_and_validate_source(test_source, test_target) == mock_response.content
+    assert fetch_and_validate_source(test_source, test_target) == mock_response.text
 
 
 @patch("requests.get")
@@ -46,7 +46,7 @@ def test_fetch_and_validate_source_when_source_unavailable(mock_get, test_source
 def test_fetch_and_validate_source_when_source_does_not_contain_target(mock_get, test_source, test_target):
     mock_response = Mock()
     mock_response.status_code = 200
-    mock_response.content = "foo"
+    mock_response.text = "foo"
     mock_get.return_value = mock_response
 
     with pytest.raises(TargetNotFoundError):