Repository: ytdl-org/youtube-dl

Issue:
error when entering username and password
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.07.28. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
- Read bugs section in FAQ: http://yt-dl.org/reporting
- Finally, put x into all relevant boxes (like this [x])
-->

- [x] I'm reporting a broken site support issue
- [x ] I've verified that I'm running youtube-dl version **2020.07.28**
- [x ] I've checked that all provided URLs are alive and playable in a browser
- [ x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x ] I've searched the bugtracker for similar bug reports including closed ones
- [x ] I've read bugs section in FAQ


## Verbose log

<!--
Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] youtube-dl version 2020.07.28
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
PASTE VERBOSE LOG HERE
```
[tim@cabra ~]$ youtube-dl --version
2020.07.28
[tim@cabra ~]$ youtube-dl -o '%(title)s.%(ext)s'   'http://mooo.peelwiki.com/dl/BillfromNorthWales/01%20John%20Peel/1978-12-26%20John%20Peel%20BBC%20Radio%201.mp3' --username 'peel' --password 'group' --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-o', u'%(title)s.%(ext)s', u'http://mooo.peelwiki.com/dl/BillfromNorthWales/01%20John%20Peel/1978-12-26%20John%20Peel%20BBC%20Radio%201.mp3', u'--username', u'PRIVATE', u'--password', u'PRIVATE', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 2.7.17 (CPython) - Linux-5.3.11-100.fc29.x86_64-x86_64-with-fedora-29-Twenty_Nine
[debug] exe versions: ffmpeg 4.0.5, ffprobe 4.0.5
[debug] Proxy map: {}
[generic] 1978-12-26 John Peel BBC Radio 1: Requesting header
WARNING: Could not send HEAD request to http://mooo.peelwiki.com/dl/BillfromNorthWales/01%20John%20Peel/1978-12-26%20John%20Peel%20BBC%20Radio%201.mp3: HTTP Error 401: Authorization Required
[generic] 1978-12-26 John Peel BBC Radio 1: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 401: Authorization Required (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[tim@cabra ~]$ 





## Description

<!--
Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

WRITE DESCRIPTION HERE


This is the first time I have had to enter a username/password combo into youtube-dl. From the browser I enter this url followed by username/password and it opens up a media player within the browser. This is the next level url

https://peel.fandom.com/wiki/1978

search for 26 December 1978 and then head to the bottom of that page for the mooo links.

I have tried a few different options with the same results - http and https. Username/password in single/double and no quotes with the same response

Regards


Discussion:
`--username` and `--password` only work for specific supported sites and not with the generic extractor.

You can download this file by adding the login credentials at the beginning of the URL and using `wget`, `curl`, your browser, etc

``` bash
$ wget 'http://peel:group@mooo.peelwiki.com/dl/BillfromNorthWales/01 John Peel/1978-12-26 John Peel BBC Radio 1.mp3'
```

`youtube-dl` also doesn't work with this kind of URL, by the way:
``` bash
$ youtube-dl -v 'http://peel:group@mooo.peelwiki.com/dl/BillfromNorthWales/01 John Peel/1978-12-26 John Peel BBC Radio 1.mp3'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'http://peel:group@mooo.peelwiki.com/dl/BillfromNorthWales/01 John Peel/1978-12-26 John Peel BBC Radio 1.mp3']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.8.5 (CPython) - Linux-5.4.50-1-lts-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.3, ffprobe 4.2.3, rtmpdump 2.4
[debug] Proxy map: {}
[generic] 1978-12-26 John Peel BBC Radio 1: Requesting header
WARNING: Could not send HEAD request to http://peel:group@mooo.peelwiki.com/dl/BillfromNorthWales/01 John Peel/1978-12-26 John Peel BBC Radio 1.mp3: nonnumeric port: 'group@mooo.peelwiki.com'
[generic] 1978-12-26 John Peel BBC Radio 1: Downloading webpage
ERROR: Unable to download webpage: nonnumeric port: 'group@mooo.peelwiki.com' (caused by InvalidURL("nonnumeric port: 'group@mooo.peelwiki.com'")); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/home/mike/.local/lib/python3.8/site-packages/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/mike/.local/lib/python3.8/site-packages/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/home/mike/.local/lib/python3.8/site-packages/youtube_dl/utils.py", line 2578, in http_open
    return self.do_open(functools.partial(
  File "/usr/lib/python3.8/urllib/request.py", line 1319, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
  File "/home/mike/.local/lib/python3.8/site-packages/youtube_dl/utils.py", line 2479, in _create_http_connection
    hc = http_class(*args, **compat_kwargs(kwargs))
  File "/usr/lib/python3.8/http/client.py", line 833, in __init__
    (self.host, self.port) = self._get_hostport(host, port)
  File "/usr/lib/python3.8/http/client.py", line 876, in _get_hostport
    raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
```
thanks for the educated response

Resolving patch:
diff --git a/youtube_dl/utils.py b/youtube_dl/utils.py
--- a/youtube_dl/utils.py
+++ b/youtube_dl/utils.py
@@ -2182,8 +2182,28 @@ def sanitize_url(url):
     return url
 
 
+def extract_basic_auth(url):
+    parts = compat_urllib_parse.urlsplit(url)
+    if parts.username is None:
+        return url, None
+    url = compat_urllib_parse.urlunsplit(parts._replace(netloc=(
+        parts.hostname if parts.port is None
+        else '%s:%d' % (parts.hostname, parts.port))))
+    auth_payload = base64.b64encode(
+        ('%s:%s' % (parts.username, parts.password or '')).encode('utf-8'))
+    return url, 'Basic {0}'.format(auth_payload.decode('ascii'))
+
+
 def sanitized_Request(url, *args, **kwargs):
-    return compat_urllib_request.Request(escape_url(sanitize_url(url)), *args, **kwargs)
+    url, auth_header = extract_basic_auth(escape_url(sanitize_url(url)))
+    if auth_header is not None:
+        headers = args[1] if len(args) > 1 else kwargs.get('headers')
+        headers = headers or {}
+        headers['Authorization'] = auth_header
+        if len(args) <= 1 and kwargs.get('headers') is None:
+            kwargs['headers'] = headers
+            kwargs = compat_kwargs(kwargs)
+    return compat_urllib_request.Request(url, *args, **kwargs)
 
 
 def expand_path(s):


Test patch:
diff --git a/test/test_utils.py b/test/test_utils.py
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -81,6 +81,7 @@
     sanitize_filename,
     sanitize_path,
     sanitize_url,
+    sanitized_Request,
     shell_quote,
     smuggle_url,
     str_or_none,
@@ -255,6 +256,18 @@ def test_sanitize_url(self):
         self.assertEqual(sanitize_url('https://foo.bar'), 'https://foo.bar')
         self.assertEqual(sanitize_url('foo bar'), 'foo bar')
 
+    def test_sanitized_Request(self):
+        self.assertFalse(sanitized_Request('http://foo.bar').has_header('Authorization'))
+        self.assertFalse(sanitized_Request('http://:foo.bar').has_header('Authorization'))
+        self.assertEqual(sanitized_Request('http://@foo.bar').get_header('Authorization'),
+                         'Basic Og==')
+        self.assertEqual(sanitized_Request('http://:pass@foo.bar').get_header('Authorization'),
+                         'Basic OnBhc3M=')
+        self.assertEqual(sanitized_Request('http://user:@foo.bar').get_header('Authorization'),
+                         'Basic dXNlcjo=')
+        self.assertEqual(sanitized_Request('http://user:pass@foo.bar').get_header('Authorization'),
+                         'Basic dXNlcjpwYXNz')
+
     def test_expand_path(self):
         def env(var):
             return '%{0}%'.format(var) if sys.platform == 'win32' else '${0}'.format(var)