Repository: awdeorio/mailmerge

Issue:
Chardet specifies wrong encoding with emoji
## Tl;dr
It looks like `chardet` doesn't always get the encoding right — in a message containing emoji, `chardet` reported (with low confidence) that the message was encoded in [Windows-1252](https://en.wikipedia.org/wiki/Windows-1252). Python throws an exception when trying to encode a string with emoji to that charset.

I don't know if there's a good fix. Can we assume that users are responsible for specifying an encoding if it's not UTF-8? (That would remove our reliance on `chardet`.)

## The bug
Steps I followed on my Mac:
1. Use the following `mailmerge_template.txt`:
```
TO: {{email}}
SUBJECT: Testing mailmerge
FROM: My Self <myself@mydomain.com>

Hi 😀
```
2. Run `mailmerge --dry-run --limit 1`

I received this error:
```
...
File "/Users/seshrs/Documents/Git/mailmerge/mailmerge/template_message.py", line 76, in _transform_encoding
    part.set_charset(encoding)
  File "/Users/seshrs/Documents/Git/mailmerge/env/lib/python3.7/site-packages/future/backports/email/message.py", line 322, in set_charset
    self._payload = charset.body_encode(self._payload)
  File "/Users/seshrs/Documents/Git/mailmerge/env/lib/python3.7/site-packages/future/backports/email/charset.py", line 403, in body_encode
    string = string.encode(self.output_charset)
  File "/usr/local/bin/../Cellar/python/3.7.4_1/bin/../Frameworks/Python.framework/Versions/3.7/lib/python3.7/encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f600' in position 3: character maps to <undefined>
```

## The cause
I set a breakpoint in the `_transform_encoding` function:
https://github.com/awdeorio/mailmerge/blob/3a78fbd4372cd67e85cf8930288fbd0e2f2adfc1/mailmerge/template_message.py#L66-L74

And printed the contents of `detected`:
```
(Pdb++) detected
{'encoding': 'Windows-1252', 'confidence': 0.5334615384615384, 'language': ''}
```

I had expected 'utf-8', not 'Windows-1252'. To confirm this was the issue, I tried executing the following in Python:
```
>>> "hi 😀".encode('Windows-1252')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f600' in position 3: character maps to <undefined>
```

(The call to `part.set_charset(encoding)` eventually reaches [this line](https://github.com/PythonCharmers/python-future/blob/01a1d313b0e17580f31e3d02a2225d7b6f471aff/src/future/backports/email/charset.py#L408) in the `python-future` code that executes something like the above.)

Discussion:

Patch:
diff --git a/mailmerge/template_message.py b/mailmerge/template_message.py
index a442cfc..edee1c5 100644
--- a/mailmerge/template_message.py
+++ b/mailmerge/template_message.py
@@ -14,7 +14,6 @@ import future.backports.email.utils
 import future.backports.email.generator
 import markdown
 import jinja2
-import chardet
 from . import utils
 
 # Python 2 pathlib support requires backport
@@ -61,6 +60,7 @@ class TemplateMessage(object):
     def render(self, context):
         """Return rendered message object."""
         raw_message = self.template.render(context)
+        self._message = email.message_from_string(raw_message)
         self._transform_encoding(raw_message)
         self._transform_recipients()
         self._transform_markdown()
@@ -73,9 +73,7 @@ class TemplateMessage(object):
 
     def _transform_encoding(self, raw_message):
         """Detect and set character encoding."""
-        self._message = email.parser.Parser().parsestr(raw_message)
-        detected = chardet.detect(bytearray(raw_message, "utf-8"))
-        encoding = detected["encoding"]
+        encoding = "us-ascii" if is_ascii(raw_message) else "utf-8"
         for part in self._message.walk():
             if part.get_content_maintype() == 'multipart':
                 continue
@@ -206,3 +204,10 @@ class TemplateMessage(object):
             raise utils.MailmergeError("Attachment not found: {}".format(path))
 
         return path
+
+
+def is_ascii(string):
+    """Return True is string contains only is us-ascii encoded characters."""
+    def is_ascii_char(char):
+        return 0 <= ord(char) <= 127
+    return all(is_ascii_char(char) for char in string)
diff --git a/setup.py b/setup.py
index 2160d8e..7886b70 100644
--- a/setup.py
+++ b/setup.py
@@ -22,7 +22,6 @@ setup(
     keywords=["mail merge", "mailmerge", "email"],
     install_requires=[
         "backports.csv;python_version<'3.0'",
-        "chardet",
         "click",
         "configparser",
 


Test patch:
diff --git a/tests/test_template_message.py b/tests/test_template_message.py
index 0baa86b..942a010 100644
--- a/tests/test_template_message.py
+++ b/tests/test_template_message.py
@@ -244,3 +244,70 @@ def test_utf8_database():
     payload = message.get_payload()
     payload = message.get_payload().replace("\n", "")
     assert payload == 'SGksIExhyJ1hbW9uLAoKWW91ciBudW1iZXIgaXMgMTcu'
+
+
+def test_emoji():
+    """Verify emoji are encoded."""
+    template_message = mailmerge.template_message.TemplateMessage(
+        template_path=utils.TESTDATA/"emoji_template.txt",
+    )
+    _, _, message = template_message.render({})
+
+    # Verify encoding
+    assert message.get_charset() == "utf-8"
+    assert message["Content-Transfer-Encoding"] == "base64"
+
+    # grinning face with smiling eyes
+    plaintext = message.get_payload().strip()
+    assert plaintext == "SGkg8J+YgA=="
+
+
+def test_emoji_markdown():
+    """Verify emoji are encoded in Markdown formatted messages."""
+    template_message = mailmerge.template_message.TemplateMessage(
+        template_path=utils.TESTDATA/"emoji_markdown_template.txt",
+    )
+    _, _, message = template_message.render({})
+
+    # Message should contain an unrendered Markdown plaintext part and a
+    # rendered Markdown HTML part
+    plaintext_part, html_part = message.get_payload()
+
+    # Verify encodings
+    assert str(plaintext_part.get_charset()) == "utf-8"
+    assert str(html_part.get_charset()) == "utf-8"
+    assert plaintext_part["Content-Transfer-Encoding"] == "base64"
+    assert html_part["Content-Transfer-Encoding"] == "base64"
+
+    # Verify content, which is base64 encoded grinning face emoji
+    plaintext = plaintext_part.get_payload().strip()
+    htmltext = html_part.get_payload().strip().replace("\n", "")
+    assert plaintext == "YGBgCmVtb2ppX3N0cmluZyA9ICDwn5iACmBgYA=="
+    assert htmltext == (
+        "PGh0bWw+PGJvZHk+PHA+"
+        "PGNvZGU+ZW1vamlfc3RyaW5nID0gIPCfmIA8L2NvZGU+"
+        "PC9wPjwvYm9keT48L2h0bWw+"
+    )
+
+
+def test_emoji_database():
+    """Verify emoji are encoded when they are substituted via template db.
+
+    The template is ASCII encoded, but after rendering the template, an emoji
+    character will substituted into the template.  The result should be a utf-8
+    encoded message.
+    """
+    template_message = mailmerge.template_message.TemplateMessage(
+        template_path=utils.TESTDATA/"emoji_database_template.txt",
+    )
+    _, _, message = template_message.render({
+        "emoji": u"\xf0\x9f\x98\x80"  # grinning face
+    })
+
+    # Verify encoding
+    assert message.get_charset() == "utf-8"
+    assert message["Content-Transfer-Encoding"] == "base64"
+
+    # Verify content
+    plaintext = message.get_payload().strip()
+    assert plaintext == "SGkgw7DCn8KYwoA="
diff --git a/tests/testdata/emoji_database_template.txt b/tests/testdata/emoji_database_template.txt
new file mode 100644
index 0000000..f2f76a6
--- /dev/null
+++ b/tests/testdata/emoji_database_template.txt
@@ -0,0 +1,5 @@
+TO: test@test.com
+SUBJECT: Testing mailmerge
+FROM: test@test.com
+
+Hi {{emoji}}
diff --git a/tests/testdata/emoji_markdown_template.txt b/tests/testdata/emoji_markdown_template.txt
new file mode 100644
index 0000000..78430e5
--- /dev/null
+++ b/tests/testdata/emoji_markdown_template.txt
@@ -0,0 +1,8 @@
+TO: test@example.com
+SUBJECT: Testing mailmerge
+FROM: test@example.com
+CONTENT-TYPE: text/markdown
+
+```
+emoji_string =  😀
+```
diff --git a/tests/testdata/emoji_template.txt b/tests/testdata/emoji_template.txt
new file mode 100644
index 0000000..7f13eda
--- /dev/null
+++ b/tests/testdata/emoji_template.txt
@@ -0,0 +1,5 @@
+TO: test@test.com
+SUBJECT: Testing mailmerge
+FROM: test@test.com
+
+Hi 😀