Repository: zulip/zulip

Issue:
URL preview garbled Chinese
Version: Self hosted 3.3

## Scenario
Server with `URL Preview` enabled
posting this link: https://newtalk.tw/news/view/2020-12-03/503169
Site uses html lang tag `lang=“zh-Hant-TW”`

## Expected Behavior
A URL leading to a Chinese website should produce a Chinese preview.

## Actual Behavior
The preview contains garbled characters like [mojibake](https://revealingerrors.com/mojibake).
> Or here's something equivalent to "interpreting it as if it were ISO-8859-1" which might turn out to be a better description of what's really happening in this bug: taking the bytes of the UTF-8 encoding, and interpreting each single byte as a Unicode codepoint.

![Screen Shot 2020-12-04 at 4 36 22 PM](https://user-images.githubusercontent.com/15863509/101258381-1b091b80-375d-11eb-9710-8dcba9fa3a65.png)

```
# Actual content of header meta title
蘋果AirDrop技術爆大漏洞！駭客可遠端操控iPhone | 科技 | 新頭殼 Newtalk

# Actual Content of header meta description
Google Project Zero的網路安全研究人員Ian Beer日前揭露了iPhone過去的重大漏洞，破解蘋果應用在AirDrop的相關技術，使得有心人士得以透過特殊裝置遠段操控附近的iPhone，所幸蘋果已經在今年5月修補了這個iOS漏洞。Beer所發現的系統漏洞存在於蘋果2014年引進的

# Garbled preview version in zulip
èæAirDropæè¡çå¤§æ¼æ´ï¼é§­å®¢å¯é ç«¯ææ§iPhone | ç§æ | æ°é ­æ®¼ Newtalk
Google Project Zeroçç¶²è·¯å®å¨ç ç©¶äººå¡Ian Beeræ¥åæ­é²äºiPhoneéå»çéå¤§æ¼æ´ï¼ç ´è§£èææç¨å¨AirDropçç¸éæè¡ï¼ä½¿å¾æå¿äººå£«å¾ä»¥ééç¹æ®è£ç½®é æ®µææ§éè¿çiPhoneï¼æå¹¸èæå·²ç¶å¨ä»å¹´5æä¿®è£äºéåiOSæ¼æ´ãBeeræç¼ç¾çç³»çµ±æ¼æ´å­å¨æ¼èæ2014å¹´å¼é²ç
```

## Possible Issue
Spot-checking just the first couple of characters here: these mojibake are consistent with getting something that's actually UTF-8 and interpreting it as if it were ISO-8859-1. Specifically:

> 蘋

This first character is e8 98 8b in UTF-8:
```
$ unicode -s 蘋 --long
U+860B CJK UNIFIED IDEOGRAPH-860B
UTF-8: e8 98 8b UTF-16BE: 860b Decimal: &#34315; Octal: \0103013
蘋
…
```
and interpreting the byte e8 in ISO-8859-1, you get è:
```
$ unicode u+00e8
è U+00E8 LATIN SMALL LETTER E WITH GRAVE
```
## Related Code
https://github.com/zulip/zulip/blob/0d6c771baf9ea0ebd8174c927e50709d59da0f64/zerver/lib/url_preview/preview.py

## Related issue chat
https://chat.zulip.org/#narrow/stream/9-issues/topic/URL.20preview.20Chinese.20Jumbled


Discussion:
@zulipbot add "bug" "area: general UI" 
https://chat.zulip.org/#narrow/stream/9-issues/topic/URL.20preview.20Chinese.20Jumbled/near/1076820 has the bug pointed out by Greg. See it before working on this issue.
Copying here a bit of partial debugging I mentioned in the linked chat thread:

This looks like [mojibake](https://revealingerrors.com/mojibake), and specifically the kind of mojibake you get when taking some data that's actually encoded in UTF-8 and interpreting it as if it were encoded in ISO-8859-1.

Or equivalently: it looks like exactly what we'd get if the data is actually encoded in UTF-8 and we interpret each single byte of it as if it were a Unicode codepoint.

I took a quick look at the code involved and don't see an obvious spot where the bug might be. (I'm not familiar with this area of our code.) The next step is probably for someone to take the given reproducer and do some debugging to find out exactly where in the code the bad data (mis-decoded data) comes from.

I wonder if this has anything to do with beautiful soup (web crawler) used to extract data from the page
Specifically Beautiful soup seems to "guess" the encoding of the source document. 
https://www.crummy.com/software/BeautifulSoup/bs4/doc/#encodings
No, the problem is the opposite: we prevent Beautiful Soup from detecting the encoding by passing `str` instead of `bytes`. The pages you’ve referenced don’t send a charset in their `Content-Type`, so we need to let Beautiful Soup detect the encoding from the `<meta>` tags in the page.

Resolving patch:
diff --git a/zerver/lib/url_preview/parsers/base.py b/zerver/lib/url_preview/parsers/base.py
--- a/zerver/lib/url_preview/parsers/base.py
+++ b/zerver/lib/url_preview/parsers/base.py
@@ -1,13 +1,17 @@
-from typing import Any
+import cgi
+from typing import Any, Optional
 
 
 class BaseParser:
-    def __init__(self, html_source: str) -> None:
+    def __init__(self, html_source: bytes, content_type: Optional[str]) -> None:
         # We import BeautifulSoup here, because it's not used by most
         # processes in production, and bs4 is big enough that
         # importing it adds 10s of milliseconds to manage.py startup.
         from bs4 import BeautifulSoup
-        self._soup = BeautifulSoup(html_source, "lxml")
+        charset = None
+        if content_type is not None:
+            charset = cgi.parse_header(content_type)[1].get("charset")
+        self._soup = BeautifulSoup(html_source, "lxml", from_encoding=charset)
 
     def extract_data(self) -> Any:
         raise NotImplementedError()
diff --git a/zerver/lib/url_preview/preview.py b/zerver/lib/url_preview/preview.py
--- a/zerver/lib/url_preview/preview.py
+++ b/zerver/lib/url_preview/preview.py
@@ -91,12 +91,16 @@ def get_link_embed_data(url: str,
 
     response = requests.get(mark_sanitized(url), stream=True, headers=HEADERS, timeout=TIMEOUT)
     if response.ok:
-        og_data = OpenGraphParser(response.text).extract_data()
+        og_data = OpenGraphParser(
+            response.content, response.headers.get("Content-Type")
+        ).extract_data()
         for key in ['title', 'description', 'image']:
             if not data.get(key) and og_data.get(key):
                 data[key] = og_data[key]
 
-        generic_data = GenericParser(response.text).extract_data() or {}
+        generic_data = GenericParser(
+            response.content, response.headers.get("Content-Type")
+        ).extract_data() or {}
         for key in ['title', 'description', 'image']:
             if not data.get(key) and generic_data.get(key):
                 data[key] = generic_data[key]


Test patch:
diff --git a/zerver/lib/test_helpers.py b/zerver/lib/test_helpers.py
--- a/zerver/lib/test_helpers.py
+++ b/zerver/lib/test_helpers.py
@@ -314,6 +314,7 @@ def get_host(self) -> str:
 
 class MockPythonResponse:
     def __init__(self, text: str, status_code: int, headers: Optional[Dict[str, str]]=None) -> None:
+        self.content = text.encode()
         self.text = text
         self.status_code = status_code
         if headers is None:
diff --git a/zerver/tests/test_link_embed.py b/zerver/tests/test_link_embed.py
--- a/zerver/tests/test_link_embed.py
+++ b/zerver/tests/test_link_embed.py
@@ -136,7 +136,7 @@ def test_autodiscovered_oembed_xml_format_html(self) -> None:
 
 class OpenGraphParserTestCase(ZulipTestCase):
     def test_page_with_og(self) -> None:
-        html = """<html>
+        html = b"""<html>
           <head>
           <meta property="og:title" content="The Rock" />
           <meta property="og:type" content="video.movie" />
@@ -146,14 +146,14 @@ def test_page_with_og(self) -> None:
           </head>
         </html>"""
 
-        parser = OpenGraphParser(html)
+        parser = OpenGraphParser(html, "text/html; charset=UTF-8")
         result = parser.extract_data()
         self.assertIn('title', result)
         self.assertEqual(result['title'], 'The Rock')
         self.assertEqual(result.get('description'), 'The Rock film')
 
     def test_page_with_evil_og_tags(self) -> None:
-        html = """<html>
+        html = b"""<html>
           <head>
           <meta property="og:title" content="The Rock" />
           <meta property="og:type" content="video.movie" />
@@ -165,7 +165,7 @@ def test_page_with_evil_og_tags(self) -> None:
           </head>
         </html>"""
 
-        parser = OpenGraphParser(html)
+        parser = OpenGraphParser(html, "text/html; charset=UTF-8")
         result = parser.extract_data()
         self.assertIn('title', result)
         self.assertEqual(result['title'], 'The Rock')
@@ -173,9 +173,30 @@ def test_page_with_evil_og_tags(self) -> None:
         self.assertEqual(result.get('oembed'), None)
         self.assertEqual(result.get('html'), None)
 
+    def test_charset_in_header(self) -> None:
+        html = """<html>
+          <head>
+            <meta property="og:title" content="中文" />
+          </head>
+        </html>""".encode("big5")
+        parser = OpenGraphParser(html, "text/html; charset=Big5")
+        result = parser.extract_data()
+        self.assertEqual(result["title"], "中文")
+
+    def test_charset_in_meta(self) -> None:
+        html = """<html>
+          <head>
+            <meta content-type="text/html; charset=Big5" />
+            <meta property="og:title" content="中文" />
+          </head>
+        </html>""".encode("big5")
+        parser = OpenGraphParser(html, "text/html")
+        result = parser.extract_data()
+        self.assertEqual(result["title"], "中文")
+
 class GenericParserTestCase(ZulipTestCase):
     def test_parser(self) -> None:
-        html = """
+        html = b"""
           <html>
             <head><title>Test title</title></head>
             <body>
@@ -184,13 +205,13 @@ def test_parser(self) -> None:
             </body>
           </html>
         """
-        parser = GenericParser(html)
+        parser = GenericParser(html, "text/html; charset=UTF-8")
         result = parser.extract_data()
         self.assertEqual(result.get('title'), 'Test title')
         self.assertEqual(result.get('description'), 'Description text')
 
     def test_extract_image(self) -> None:
-        html = """
+        html = b"""
           <html>
             <body>
                 <h1>Main header</h1>
@@ -202,14 +223,14 @@ def test_extract_image(self) -> None:
             </body>
           </html>
         """
-        parser = GenericParser(html)
+        parser = GenericParser(html, "text/html; charset=UTF-8")
         result = parser.extract_data()
         self.assertEqual(result.get('title'), 'Main header')
         self.assertEqual(result.get('description'), 'Description text')
         self.assertEqual(result.get('image'), 'http://test.com/test.jpg')
 
     def test_extract_description(self) -> None:
-        html = """
+        html = b"""
           <html>
             <body>
                 <div>
@@ -220,22 +241,22 @@ def test_extract_description(self) -> None:
             </body>
           </html>
         """
-        parser = GenericParser(html)
+        parser = GenericParser(html, "text/html; charset=UTF-8")
         result = parser.extract_data()
         self.assertEqual(result.get('description'), 'Description text')
 
-        html = """
+        html = b"""
           <html>
             <head><meta name="description" content="description 123"</head>
             <body></body>
           </html>
         """
-        parser = GenericParser(html)
+        parser = GenericParser(html, "text/html; charset=UTF-8")
         result = parser.extract_data()
         self.assertEqual(result.get('description'), 'description 123')
 
-        html = "<html><body></body></html>"
-        parser = GenericParser(html)
+        html = b"<html><body></body></html>"
+        parser = GenericParser(html, "text/html; charset=UTF-8")
         result = parser.extract_data()
         self.assertIsNone(result.get('description'))