Repository: ytdl-org/youtube-dl

Issue:
utils.clean_html produces different output on Python 2.x and 3.x
## Please follow the guide below

- You will be asked some questions and requested to provide some information, please read them **carefully** and answer honestly
- Put an `x` into all the boxes [ ] relevant to your *issue* (like that [x])
- Use *Preview* tab to see how your issue will actually look like

---

### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2017.04.26*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [x] I've **verified** and **I assure** that I'm running youtube-dl **2017.04.26**

### Before submitting an *issue* make sure you have:
- [x] At least skimmed through [README](https://github.com/rg3/youtube-dl/blob/master/README.md) and **most notably** [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
- [x] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones

### What is the purpose of your *issue*?
- [x] Bug report (encountered problems with youtube-dl)
- [ ] Site support request (request for adding support for a new site)
- [ ] Feature request (request for a new functionality)
- [ ] Question
- [ ] Other

---

In Python 3.x `\s` matches non-breaking space (`\xa0`), in Python 2.x it doesn't. This causes different output and fails in tests, for example `test_ArchiveOrg_1`:

Python 3.6:
```
F:\GitHub\youtube-dl>python test\test_download.py TestDownload.test_ArchiveOrg_1
[archive.org] Cops1922: Downloading webpage
[archive.org] Cops1922: Downloading JSON metadata
[info] Writing video description metadata as JSON to: test_ArchiveOrg_1_Cops1922.info.json
[debug] Invoking downloader on 'https://archive.org/download/Cops1922/Cops-v2.mp4'
[download] Destination: test_ArchiveOrg_1_Cops1922.mp4
[download] 100% of 10.00KiB in 00:00
.
----------------------------------------------------------------------
Ran 1 test in 4.415s

OK
```
Python 2.7:
```
F:\GitHub\youtube-dl>python2 test\test_download.py TestDownload.test_ArchiveOrg_1
[archive.org] Cops1922: Downloading webpage
[archive.org] Cops1922: Downloading JSON metadata
[info] Writing video description metadata as JSON to: test_ArchiveOrg_1_Cops1922.info.json
[debug] Invoking downloader on u'https://archive.org/download/Cops1922/Cops-v2.mp4'
[download] Destination: test_ArchiveOrg_1_Cops1922.mp4
[download] 100% of 10.00KiB in 00:00
F
======================================================================
FAIL: test_ArchiveOrg_1 (__main__.TestDownload):
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test\test_download.py", line 210, in test_template
    expect_info_dict(self, tc_res_dict, tc.get('info_dict', {}))
  File "F:\GitHub\youtube-dl\test\helper.py", line 177, in expect_info_dict
    expect_dict(self, got_dict, expected_dict)
  File "F:\GitHub\youtube-dl\test\helper.py", line 173, in expect_dict
    expect_value(self, got, expected, info_field)
  File "F:\GitHub\youtube-dl\test\helper.py", line 167, in expect_value
    'Invalid value for field %s, expected %r, got %r' % (field, expected, got))
AssertionError: Invalid value for field description, expected u'md5:89e7c77bf5d965dd5c0372cfb49470f6', got u'md5:186aee7106d70bfa6dff7ea443855c7c'

----------------------------------------------------------------------
Ran 1 test in 4.190s

FAILED (failures=1)
```

Original from `clean_html`:
```python
    html = re.sub(r'\s*<\s*br\s*/?\s*>\s*', '\n', html)
    html = re.sub(r'<\s*/\s*p\s*>\s*<\s*p[^>]*>', '\n', html)
```

Solution 1:
```python
    html = html.replace('\xa0', ' ')
    # no change below
    html = re.sub(r'\s*<\s*br\s*/?\s*>\s*', '\n', html)
    html = re.sub(r'<\s*/\s*p\s*>\s*<\s*p[^>]*>', '\n', html)
```
*may break some other tests*

Solution 2a:
```python
    html = re.sub(r'[\s\xa0]*<\s*br\s*/?\s*>[\s\xa0]*', '\n', html)
    html = re.sub(r'<\s*/\s*p\s*>[\s\xa0]*<\s*p[^>]*>', '\n', html)
```
or 2b:
```python
    html = re.sub('[\\s\xa0]*<\\s*br\\s*/?\\s*>[\\s\xa0]*', '\n', html)
    html = re.sub('<\\s*/\\s*p\\s*>[\\s\xa0]*<\\s*p[^>]*>', '\n', html)
```
not really sure which one of them is correct because both of them work.

Which solution is better?


Discussion:
> In Python 3.x \s matches non-breaking space (\xa0), in Python 2.x it doesn't.

That's because in Python 3 unicode patterns are used by default. You can force Unicode on Python 2 or ASCII matching on Python 3.

```
$ python3 -c 'import re; print(re.match("(?a)\s", "\xa0"))'
None
$ python2 -c 'import re; print(re.match("(?u)\s", "\xa0"))'
<_sre.SRE_Match object at 0x7f325b186578>
```

Now here comes the next question: which one should be used? The HTML5 standard [3] is closer to the ASCII way, so ```(?a)``` is more reasonable.

References:
[1] Unicode spaces: https://github.com/python/cpython/blob/master/Objects/unicodetype_db.h#L5741
[2] ASCII spaces: https://github.com/python/cpython/blob/master/Python/pyctype.c#L5
[3] https://www.w3.org/TR/html5/infrastructure.html#space-character
`\s` is supposed to match whitespace characters, not just spaces. And as [3] says:

_The **White_Space characters** are those that have the Unicode property "White_Space" in the Unicode [`PropList.txt`](http://ftp.unicode.org/Public/UNIDATA/PropList.txt) data file._

and inside that file:

    00A0          ; White_Space # Zs       NO-BREAK SPACE

so I guess `(?u)`?
In https://www.w3.org/TR/html5/syntax.html#start-tags, a start tag in HTML uses "space characters" rather than "White_Space characters". Anyway many websites doesn't obey HTML standards, so it's OK as long as the result is consistent on Python 2 and 3.
Also: `(?u)` can be used in Python 3, `(?a)` can't in Python 2.

Resolving patch:
diff --git a/youtube_dl/utils.py b/youtube_dl/utils.py
--- a/youtube_dl/utils.py
+++ b/youtube_dl/utils.py
@@ -421,8 +421,8 @@ def clean_html(html):
 
     # Newline vs <br />
     html = html.replace('\n', ' ')
-    html = re.sub(r'\s*<\s*br\s*/?\s*>\s*', '\n', html)
-    html = re.sub(r'<\s*/\s*p\s*>\s*<\s*p[^>]*>', '\n', html)
+    html = re.sub(r'(?u)\s*<\s*br\s*/?\s*>\s*', '\n', html)
+    html = re.sub(r'(?u)<\s*/\s*p\s*>\s*<\s*p[^>]*>', '\n', html)
     # Strip html tags
     html = re.sub('<.*?>', '', html)
     # Replace html entities


Test patch:
diff --git a/test/test_utils.py b/test/test_utils.py
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -899,6 +899,7 @@ def test_extract_attributes(self):
     def test_clean_html(self):
         self.assertEqual(clean_html('a:\nb'), 'a: b')
         self.assertEqual(clean_html('a:\n   "b"'), 'a:    "b"')
+        self.assertEqual(clean_html('a<br>\xa0b'), 'a\nb')
 
     def test_intlist_to_bytes(self):
         self.assertEqual(