Repository: ytdl-org/youtube-dl

Issue:
Response file object (returned by `InfoExtractor._request_webpage`) may be closed for failed requests (matching `expected_status`) on Python 3.4.1+
### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2018.08.04*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [x] I've **verified** and **I assure** that I'm running youtube-dl **2018.08.04**

### Before submitting an *issue* make sure you have:
- [x] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
- [x] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
- [x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

### What is the purpose of your *issue*?
- [x] Bug report (encountered problems with youtube-dl)
- [ ] Site support request (request for adding support for a new site)
- [ ] Feature request (request for a new functionality)
- [ ] Question
- [ ] Other

---

Since [bpo-15002](https://bugs.python.org/issue15002) (introduced in [Python 3.4.1](https://docs.python.org/3.4/whatsnew/changelog.html#id42)), `HTTPError`s close their `fp` when the error's destroyed. The current implementation of `InfoExtractor._request_webpage` (used by `InfoExtractor._download_webpage_handle` and in turn by `InfoExtractor._download_`{`webpage`, `xml`, and `json`}) accommodates for `expected_status` by catching `HTTPError`s and returning this `fp`. Unfortunately, this means subsequent reads against this file object by the caller are unreliable.

  * If `fp` is an instance of `http.client.HTTPResponse`, we [read](https://github.com/rg3/youtube-dl/blob/2018.10.05/youtube_dl/extractor/common.py#L686) out an empty response body.
  * If `fp` is an instance of `urllib.response.addinfourl` (for when youtube-dl handles [gzip](https://github.com/rg3/youtube-dl/blob/2018.10.05/youtube_dl/utils.py#L1043) and [deflate](https://github.com/rg3/youtube-dl/blob/2018.10.05/youtube_dl/utils.py#L1049) responses), the attempted [read](https://github.com/rg3/youtube-dl/blob/2018.10.05/youtube_dl/extractor/common.py#L686) raises a `ValueError: I/O operation on closed file` exception, as demonstrated in #17447.
  * On Windows, `tempfile._TemporaryFileCloser` omits an implementation of `__del__` that would close `fp`, so reads return successfully. This platform inconsistency has been reported as [bpo-34958](https://bugs.python.org/issue34958).

Fortunately, the number of extractors that make use of `expected_status` is small; as of 2018.08.04, it's just `bbc`, `lynda`, `markiza`, and `twitch`.

Issue encountered whilst debugging [reports](https://github.com/rg3/youtube-dl/issues/17116#issuecomment-411159316) of problems running @bato3's [fix](https://github.com/rg3/youtube-dl/compare/master...bato3:crunchroll2) for #17116 on Python 3.7.

Response file object (returned by `InfoExtractor._request_webpage`) may be closed for failed requests (matching `expected_status`) on Python 3.4.1+
### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2018.08.04*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [x] I've **verified** and **I assure** that I'm running youtube-dl **2018.08.04**

### Before submitting an *issue* make sure you have:
- [x] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
- [x] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
- [x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

### What is the purpose of your *issue*?
- [x] Bug report (encountered problems with youtube-dl)
- [ ] Site support request (request for adding support for a new site)
- [ ] Feature request (request for a new functionality)
- [ ] Question
- [ ] Other

---

Since [bpo-15002](https://bugs.python.org/issue15002) (introduced in [Python 3.4.1](https://docs.python.org/3.4/whatsnew/changelog.html#id42)), `HTTPError`s close their `fp` when the error's destroyed. The current implementation of `InfoExtractor._request_webpage` (used by `InfoExtractor._download_webpage_handle` and in turn by `InfoExtractor._download_`{`webpage`, `xml`, and `json`}) accommodates for `expected_status` by catching `HTTPError`s and returning this `fp`. Unfortunately, this means subsequent reads against this file object by the caller are unreliable.

  * If `fp` is an instance of `http.client.HTTPResponse`, we [read](https://github.com/rg3/youtube-dl/blob/2018.10.05/youtube_dl/extractor/common.py#L686) out an empty response body.
  * If `fp` is an instance of `urllib.response.addinfourl` (for when youtube-dl handles [gzip](https://github.com/rg3/youtube-dl/blob/2018.10.05/youtube_dl/utils.py#L1043) and [deflate](https://github.com/rg3/youtube-dl/blob/2018.10.05/youtube_dl/utils.py#L1049) responses), the attempted [read](https://github.com/rg3/youtube-dl/blob/2018.10.05/youtube_dl/extractor/common.py#L686) raises a `ValueError: I/O operation on closed file` exception, as demonstrated in #17447.
  * On Windows, `tempfile._TemporaryFileCloser` omits an implementation of `__del__` that would close `fp`, so reads return successfully. This platform inconsistency has been reported as [bpo-34958](https://bugs.python.org/issue34958).

Fortunately, the number of extractors that make use of `expected_status` is small; as of 2018.08.04, it's just `bbc`, `lynda`, `markiza`, and `twitch`.

Issue encountered whilst debugging [reports](https://github.com/rg3/youtube-dl/issues/17116#issuecomment-411159316) of problems running @bato3's [fix](https://github.com/rg3/youtube-dl/compare/master...bato3:crunchroll2) for #17116 on Python 3.7.

[test_InfoExtractor] Add test case for #17195
### Before submitting a *pull request* make sure you have:
- [x] At least skimmed through [adding new extractor tutorial](https://github.com/rg3/youtube-dl#adding-support-for-a-new-site) and [youtube-dl coding conventions](https://github.com/rg3/youtube-dl#youtube-dl-coding-conventions) sections
- [x] [Searched](https://github.com/rg3/youtube-dl/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [x] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)

### In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
- [x] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)

### What is the purpose of your *pull request*?
- [x] Bug fix
- [ ] Improvement
- [ ] New extractor
- [ ] New feature

---

### Description of your *pull request* and other information

The purpose of this PR is to get Travis to run the new `test_response_with_expected_status_returns_content (test.test_InfoExtractor.TestInfoExtractor)`, which should reliably fail on Python 3.4.1+ as described in #17195.

This PR should _not_ be merged; its changes will be incorporated into the fix opened in #17199.


Discussion:
I don't see any problem here. By using `expected_status` you treat potentially failed outcomes as normal thus you should be ready for consequences like closed connection. It's the responsibility of a client code to check whether connection was closed or not in such cases.
Also if you want error then don't use `expected_status` and catch exception instead. The whole point of `expected_status` is to simplify code in cases when success and failure are both expressed in the same way. For example, when `_download_json` always returns JSON (e.g. for 404 and for 403) so that in 403 scenario you don't need to catch `HTTPError`, read output and parse it in client code.
@dstftw, we're not talking about a closed connection here, we're talking about the response body being inaccessible, which defeats the point of `expected_status`. The expectation is that if you call `_download_webpage` with `expected_status`, you'll get the response body back if the request was successful _or_ if the response code matches `expected_status`. But this is not the case; on _expected_ errors, the response file object returned by `_request_webpage` will be closed by the time `_download_webpage_handle` tries to extract its contents with `_webpage_read_content`.

So unless you're saying that in usages like `page = self._download_webpage('https://httpbin.org/status/418', None, expected_status=418)`, we're to expect an empty string if the request returns a 418, this is definitely a bug.
I don't see any problem here. By using `expected_status` you treat potentially failed outcomes as normal thus you should be ready for consequences like closed connection. It's the responsibility of a client code to check whether connection was closed or not in such cases.
Also if you want error then don't use `expected_status` and catch exception instead. The whole point of `expected_status` is to simplify code in cases when success and failure are both expressed in the same way. For example, when `_download_json` always returns JSON (e.g. for 404 and for 403) so that in 403 scenario you don't need to catch `HTTPError`, read output and parse it in client code.
@dstftw, we're not talking about a closed connection here, we're talking about the response body being inaccessible, which defeats the point of `expected_status`. The expectation is that if you call `_download_webpage` with `expected_status`, you'll get the response body back if the request was successful _or_ if the response code matches `expected_status`. But this is not the case; on _expected_ errors, the response file object returned by `_request_webpage` will be closed by the time `_download_webpage_handle` tries to extract its contents with `_webpage_read_content`.

So unless you're saying that in usages like `page = self._download_webpage('https://httpbin.org/status/418', None, expected_status=418)`, we're to expect an empty string if the request returns a 418, this is definitely a bug.


Resolving patch:
diff --git a/youtube_dl/extractor/common.py b/youtube_dl/extractor/common.py
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@@ -605,6 +605,11 @@ def _request_webpage(self, url_or_request, video_id, note=None, errnote=None, fa
         except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
             if isinstance(err, compat_urllib_error.HTTPError):
                 if self.__can_accept_status_code(err, expected_status):
+                    # Retain reference to error to prevent file object from
+                    # being closed before it can be read. Works around the
+                    # effects of <https://bugs.python.org/issue15002>
+                    # introduced in Python 3.4.1.
+                    err.fp._error = err
                     return err.fp
 
             if errnote is False:


Test patch:
diff --git a/test/helper.py b/test/helper.py
--- a/test/helper.py
+++ b/test/helper.py
@@ -7,6 +7,7 @@
 import os.path
 import re
 import types
+import ssl
 import sys
 
 import youtube_dl.extractor
@@ -244,3 +245,12 @@ def _report_warning(w):
             real_warning(w)
 
     ydl.report_warning = _report_warning
+
+
+def http_server_port(httpd):
+    if os.name == 'java' and isinstance(httpd.socket, ssl.SSLSocket):
+        # In Jython SSLSocket is not a subclass of socket.socket
+        sock = httpd.socket.sock
+    else:
+        sock = httpd.socket
+    return sock.getsockname()[1]
diff --git a/test/test_InfoExtractor.py b/test/test_InfoExtractor.py
--- a/test/test_InfoExtractor.py
+++ b/test/test_InfoExtractor.py
@@ -9,11 +9,30 @@
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import FakeYDL, expect_dict, expect_value
-from youtube_dl.compat import compat_etree_fromstring
+from test.helper import FakeYDL, expect_dict, expect_value, http_server_port
+from youtube_dl.compat import compat_etree_fromstring, compat_http_server
 from youtube_dl.extractor.common import InfoExtractor
 from youtube_dl.extractor import YoutubeIE, get_info_extractor
 from youtube_dl.utils import encode_data_uri, strip_jsonp, ExtractorError, RegexNotFoundError
+import threading
+
+
+TEAPOT_RESPONSE_STATUS = 418
+TEAPOT_RESPONSE_BODY = "<h1>418 I'm a teapot</h1>"
+
+
+class InfoExtractorTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
+    def log_message(self, format, *args):
+        pass
+
+    def do_GET(self):
+        if self.path == '/teapot':
+            self.send_response(TEAPOT_RESPONSE_STATUS)
+            self.send_header('Content-Type', 'text/html; charset=utf-8')
+            self.end_headers()
+            self.wfile.write(TEAPOT_RESPONSE_BODY.encode())
+        else:
+            assert False
 
 
 class TestIE(InfoExtractor):
@@ -743,6 +762,25 @@ def test_parse_xspf(self):
                 for i in range(len(entries)):
                     expect_dict(self, entries[i], expected_entries[i])
 
+    def test_response_with_expected_status_returns_content(self):
+        # Checks for mitigations against the effects of
+        # <https://bugs.python.org/issue15002> that affect Python 3.4.1+, which
+        # manifest as `_download_webpage`, `_download_xml`, `_download_json`,
+        # or the underlying `_download_webpage_handle` returning no content
+        # when a response matches `expected_status`.
+
+        httpd = compat_http_server.HTTPServer(
+            ('127.0.0.1', 0), InfoExtractorTestRequestHandler)
+        port = http_server_port(httpd)
+        server_thread = threading.Thread(target=httpd.serve_forever)
+        server_thread.daemon = True
+        server_thread.start()
+
+        (content, urlh) = self.ie._download_webpage_handle(
+            'http://127.0.0.1:%d/teapot' % port, None,
+            expected_status=TEAPOT_RESPONSE_STATUS)
+        self.assertEqual(content, TEAPOT_RESPONSE_BODY)
+
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/test/test_downloader_http.py b/test/test_downloader_http.py
--- a/test/test_downloader_http.py
+++ b/test/test_downloader_http.py
@@ -9,26 +9,16 @@
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import try_rm
+from test.helper import http_server_port, try_rm
 from youtube_dl import YoutubeDL
 from youtube_dl.compat import compat_http_server
 from youtube_dl.downloader.http import HttpFD
 from youtube_dl.utils import encodeFilename
-import ssl
 import threading
 
 TEST_DIR = os.path.dirname(os.path.abspath(__file__))
 
 
-def http_server_port(httpd):
-    if os.name == 'java' and isinstance(httpd.socket, ssl.SSLSocket):
-        # In Jython SSLSocket is not a subclass of socket.socket
-        sock = httpd.socket.sock
-    else:
-        sock = httpd.socket
-    return sock.getsockname()[1]
-
-
 TEST_SIZE = 10 * 1024
 
 
diff --git a/test/test_http.py b/test/test_http.py
--- a/test/test_http.py
+++ b/test/test_http.py
@@ -8,6 +8,7 @@
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
+from test.helper import http_server_port
 from youtube_dl import YoutubeDL
 from youtube_dl.compat import compat_http_server, compat_urllib_request
 import ssl
@@ -16,15 +17,6 @@
 TEST_DIR = os.path.dirname(os.path.abspath(__file__))
 
 
-def http_server_port(httpd):
-    if os.name == 'java' and isinstance(httpd.socket, ssl.SSLSocket):
-        # In Jython SSLSocket is not a subclass of socket.socket
-        sock = httpd.socket.sock
-    else:
-        sock = httpd.socket
-    return sock.getsockname()[1]
-
-
 class HTTPTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
     def log_message(self, format, *args):
         pass