Repository: yt-dlp/yt-dlp

Issue:
Regarding the issue that adding irregular proxies might cause some sites to fail to download
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2023.07.06** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

 When I run the program, and add the --proxy parameter without a protocol header (for example, "127.0.0.1:1080", without "http://, socks5://" etc protocol headers), certain websites start to have problems.

I have noticed that in the file yt_dlp/YoutubeDL.py, on line 4085 of the function 'build_request_director', proxies are handled in such a way that if there's no protocol header, it will automatically add the http:// protocol header.
```py
def build_request_director(self, handlers, preferences=None):
        logger = _YDLLogger(self)
        headers = self.params['http_headers'].copy()
        proxies = self.proxies.copy()
        clean_headers(headers)
        clean_proxies(proxies, headers)

        director = RequestDirector(logger=logger, verbose=self.params.get('debug_printtraffic'))
        for handler in handlers:
            director.add_handler(handler(
                logger=logger,
                headers=headers,
                cookiejar=self.cookiejar,
                proxies=proxies,
                prefer_system_certs='no-certifi' in self.params['compat_opts'],
                verify=not self.params.get('nocheckcertificate'),
                **traverse_obj(self.params, {
                    'verbose': 'debug_printtraffic',
                    'source_address': 'source_address',
                    'timeout': 'socket_timeout',
                    'legacy_ssl_support': 'legacyserverconnect',
                    'enable_file_urls': 'enable_file_urls',
                    'client_cert': {
                        'client_certificate': 'client_certificate',
                        'client_certificate_key': 'client_certificate_key',
                        'client_certificate_password': 'client_certificate_password',
                    },
                }),
            ))
        director.preferences.update(preferences or [])
        return director
```
However, I've found that in the yt_dlp/extractor/abematv.py file, when getting an instance, it does not process the proxies. This causes multiple instances to be created without adding a handler, which results in a 'url type unknown' error.
```py
def add_opener(ydl, handler):  # FIXME: Create proper API in .networking
    """Add a handler for opening URLs, like _download_webpage"""
    # https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L426
    # https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L605
    print('Adding abematv-license:// han2er22eee')
    rh = ydl._request_director.handlers['Urllib']
    if 'abematv-license' in rh._SUPPORTED_URL_SCHEMES:
        return
    
    opener = rh._get_instance(cookiejar=ydl.cookiejar, proxies=ydl.proxies)
    assert isinstance(opener, urllib.request.OpenerDirector)
    opener.add_handler(handler)
    rh._SUPPORTED_URL_SCHEMES = (*rh._SUPPORTED_URL_SCHEMES, 'abematv-license')
```

If you change the code to the following, it can fix this problem. 
```py
def add_opener(ydl, handler):  # FIXME: Create proper API in .networking
    """Add a handler for opening URLs, like _download_webpage"""
    # https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L426
    # https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L605
    print('Adding abematv-license:// han2er22eee')
    rh = ydl._request_director.handlers['Urllib']
    if 'abematv-license' in rh._SUPPORTED_URL_SCHEMES:
        return
    headers = ydl.params['http_headers'].copy()
    proxies = ydl.proxies.copy()
    clean_proxies(proxies,headers)
    opener = rh._get_instance(cookiejar=ydl.cookiejar, proxies=proxies)
    assert isinstance(opener, urllib.request.OpenerDirector)
    opener.add_handler(handler)
    rh._SUPPORTED_URL_SCHEMES = (*rh._SUPPORTED_URL_SCHEMES, 'abematv-license')
```
I'm not sure if other websites would have the same issue, but I hope this would be of some help.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
.\yt-dlp.exe -vU --proxy "127.0.0.1:1080" https://abema.tv/video/episode/90-1843_s55_p1
[debug] Command-line config: ['-vU', '--proxy', '127.0.0.1:1080', 'https://abema.tv/video/episode/90-1843_s55_p1']
[debug] Encodings: locale cp936, fs utf-8, pref cp936, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2023.09.05.203540 [d3d81cc98] (win32_dir)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.47.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {'all': '127.0.0.1:1080'}
[debug] Loaded 1864 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Available version: nightly@2023.09.05.203540, Current version: nightly@2023.09.05.203540
yt-dlp is up to date (nightly@2023.09.05.203540)
[AbemaTV] Extracting URL: https://abema.tv/video/episode/90-1843_s55_p1
[AbemaTV] Authorizing
[AbemaTV] 90-1843_s55_p1: Downloading webpage
[AbemaTV] 90-1843_s55_p1: Checking playability
[AbemaTV] 90-1843_s55_p1: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: best/bestvideo+bestaudio
[info] 90-1843_s55_p1: Downloading 1 format(s): 5300
[debug] Invoking hlsnative downloader on "https://vod-abematv.akamaized.net/program/90-1843_s55_p1/1080/playlist.m3u8?aver=1"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 758
[download] Destination: #1：新シーズン♡きみとの、ひと夏の恋の記録【しょうとあカップルデート：前編】 [90-1843_s55_p1].mp4
[debug] File locking is not supported. Proceeding without locking
[download]   0.1% of ~   2.08GiB at  586.29KiB/s ETA 01:01:45 (frag 1/758)ERROR: unable to download video data: <urlopen error unknown url type: abematv-license>
Traceback (most recent call last):
  File "yt_dlp\networking\_urllib.py", line 428, in _send
  File "urllib\request.py", line 525, in open
  File "urllib\request.py", line 547, in _open
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 1425, in unknown_open
urllib.error.URLError: <urlopen error unknown url type: abematv-license>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 3395, in process_info
  File "yt_dlp\YoutubeDL.py", line 3116, in dl
  File "yt_dlp\downloader\common.py", line 455, in download
  File "yt_dlp\downloader\hls.py", line 375, in real_download
  File "yt_dlp\downloader\fragment.py", line 526, in download_and_append_fragments
  File "yt_dlp\downloader\fragment.py", line 367, in decrypt_fragment
  File "yt_dlp\downloader\fragment.py", line 356, in _get_key
  File "yt_dlp\YoutubeDL.py", line 4059, in urlopen
  File "yt_dlp\networking\common.py", line 114, in send
  File "yt_dlp\networking\_helper.py", line 203, in wrapper
  File "yt_dlp\networking\common.py", line 325, in send
  File "yt_dlp\networking\_urllib.py", line 443, in _send
yt_dlp.networking.exceptions.TransportError: <urlopen error unknown url type: abematv-license>
```

Regarding the issue that adding irregular proxies might cause some sites to fail to download
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2023.07.06** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

 When I run the program, and add the --proxy parameter without a protocol header (for example, "127.0.0.1:1080", without "http://, socks5://" etc protocol headers), certain websites start to have problems.

I have noticed that in the file yt_dlp/YoutubeDL.py, on line 4085 of the function 'build_request_director', proxies are handled in such a way that if there's no protocol header, it will automatically add the http:// protocol header.
```py
def build_request_director(self, handlers, preferences=None):
        logger = _YDLLogger(self)
        headers = self.params['http_headers'].copy()
        proxies = self.proxies.copy()
        clean_headers(headers)
        clean_proxies(proxies, headers)

        director = RequestDirector(logger=logger, verbose=self.params.get('debug_printtraffic'))
        for handler in handlers:
            director.add_handler(handler(
                logger=logger,
                headers=headers,
                cookiejar=self.cookiejar,
                proxies=proxies,
                prefer_system_certs='no-certifi' in self.params['compat_opts'],
                verify=not self.params.get('nocheckcertificate'),
                **traverse_obj(self.params, {
                    'verbose': 'debug_printtraffic',
                    'source_address': 'source_address',
                    'timeout': 'socket_timeout',
                    'legacy_ssl_support': 'legacyserverconnect',
                    'enable_file_urls': 'enable_file_urls',
                    'client_cert': {
                        'client_certificate': 'client_certificate',
                        'client_certificate_key': 'client_certificate_key',
                        'client_certificate_password': 'client_certificate_password',
                    },
                }),
            ))
        director.preferences.update(preferences or [])
        return director
```
However, I've found that in the yt_dlp/extractor/abematv.py file, when getting an instance, it does not process the proxies. This causes multiple instances to be created without adding a handler, which results in a 'url type unknown' error.
```py
def add_opener(ydl, handler):  # FIXME: Create proper API in .networking
    """Add a handler for opening URLs, like _download_webpage"""
    # https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L426
    # https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L605
    print('Adding abematv-license:// han2er22eee')
    rh = ydl._request_director.handlers['Urllib']
    if 'abematv-license' in rh._SUPPORTED_URL_SCHEMES:
        return
    
    opener = rh._get_instance(cookiejar=ydl.cookiejar, proxies=ydl.proxies)
    assert isinstance(opener, urllib.request.OpenerDirector)
    opener.add_handler(handler)
    rh._SUPPORTED_URL_SCHEMES = (*rh._SUPPORTED_URL_SCHEMES, 'abematv-license')
```

If you change the code to the following, it can fix this problem. 
```py
def add_opener(ydl, handler):  # FIXME: Create proper API in .networking
    """Add a handler for opening URLs, like _download_webpage"""
    # https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L426
    # https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L605
    print('Adding abematv-license:// han2er22eee')
    rh = ydl._request_director.handlers['Urllib']
    if 'abematv-license' in rh._SUPPORTED_URL_SCHEMES:
        return
    headers = ydl.params['http_headers'].copy()
    proxies = ydl.proxies.copy()
    clean_proxies(proxies,headers)
    opener = rh._get_instance(cookiejar=ydl.cookiejar, proxies=proxies)
    assert isinstance(opener, urllib.request.OpenerDirector)
    opener.add_handler(handler)
    rh._SUPPORTED_URL_SCHEMES = (*rh._SUPPORTED_URL_SCHEMES, 'abematv-license')
```
I'm not sure if other websites would have the same issue, but I hope this would be of some help.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
.\yt-dlp.exe -vU --proxy "127.0.0.1:1080" https://abema.tv/video/episode/90-1843_s55_p1
[debug] Command-line config: ['-vU', '--proxy', '127.0.0.1:1080', 'https://abema.tv/video/episode/90-1843_s55_p1']
[debug] Encodings: locale cp936, fs utf-8, pref cp936, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2023.09.05.203540 [d3d81cc98] (win32_dir)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.47.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {'all': '127.0.0.1:1080'}
[debug] Loaded 1864 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Available version: nightly@2023.09.05.203540, Current version: nightly@2023.09.05.203540
yt-dlp is up to date (nightly@2023.09.05.203540)
[AbemaTV] Extracting URL: https://abema.tv/video/episode/90-1843_s55_p1
[AbemaTV] Authorizing
[AbemaTV] 90-1843_s55_p1: Downloading webpage
[AbemaTV] 90-1843_s55_p1: Checking playability
[AbemaTV] 90-1843_s55_p1: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: best/bestvideo+bestaudio
[info] 90-1843_s55_p1: Downloading 1 format(s): 5300
[debug] Invoking hlsnative downloader on "https://vod-abematv.akamaized.net/program/90-1843_s55_p1/1080/playlist.m3u8?aver=1"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 758
[download] Destination: #1：新シーズン♡きみとの、ひと夏の恋の記録【しょうとあカップルデート：前編】 [90-1843_s55_p1].mp4
[debug] File locking is not supported. Proceeding without locking
[download]   0.1% of ~   2.08GiB at  586.29KiB/s ETA 01:01:45 (frag 1/758)ERROR: unable to download video data: <urlopen error unknown url type: abematv-license>
Traceback (most recent call last):
  File "yt_dlp\networking\_urllib.py", line 428, in _send
  File "urllib\request.py", line 525, in open
  File "urllib\request.py", line 547, in _open
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 1425, in unknown_open
urllib.error.URLError: <urlopen error unknown url type: abematv-license>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 3395, in process_info
  File "yt_dlp\YoutubeDL.py", line 3116, in dl
  File "yt_dlp\downloader\common.py", line 455, in download
  File "yt_dlp\downloader\hls.py", line 375, in real_download
  File "yt_dlp\downloader\fragment.py", line 526, in download_and_append_fragments
  File "yt_dlp\downloader\fragment.py", line 367, in decrypt_fragment
  File "yt_dlp\downloader\fragment.py", line 356, in _get_key
  File "yt_dlp\YoutubeDL.py", line 4059, in urlopen
  File "yt_dlp\networking\common.py", line 114, in send
  File "yt_dlp\networking\_helper.py", line 203, in wrapper
  File "yt_dlp\networking\common.py", line 325, in send
  File "yt_dlp\networking\_urllib.py", line 443, in _send
yt_dlp.networking.exceptions.TransportError: <urlopen error unknown url type: abematv-license>
```



Discussion:
The code in the latest main branch may have this issue. However, the code from July 6, 2023, does not have this issue. I noticed that this issue may have been caused by changes to the core code.
The Abema extractor needs a complete rewrite to support the new networking framework properly, so wouldn't surprise me if there are issues.

The developer who used to maintain it is no longer online/part of this project so someone else will need to pick it up.

If anyone does want to, feel free to @ me for help on the framework side of things if need be.
> The Abema extractor needs a complete rewrite to support the new networking framework properly, so wouldn't surprise me if there are issues.
> 
> The developer who used to maintain it is no longer online/part of this project so someone else will need to pick it up.
> 
> If anyone does want to, feel free to @ me for help on the framework side of things if need be.

Okay, I understand. Unfortunately, I may not be of much help because I haven't used Python for development before. I can only provide some minor suggestions.
That said I believe your patch is the right way to fix this issue with the current hacky workaround for Abema, so feel free to open a PR with it.
The code in the latest main branch may have this issue. However, the code from July 6, 2023, does not have this issue. I noticed that this issue may have been caused by changes to the core code.
The Abema extractor needs a complete rewrite to support the new networking framework properly, so wouldn't surprise me if there are issues.

The developer who used to maintain it is no longer online/part of this project so someone else will need to pick it up.

If anyone does want to, feel free to @ me for help on the framework side of things if need be.
> The Abema extractor needs a complete rewrite to support the new networking framework properly, so wouldn't surprise me if there are issues.
> 
> The developer who used to maintain it is no longer online/part of this project so someone else will need to pick it up.
> 
> If anyone does want to, feel free to @ me for help on the framework side of things if need be.

Okay, I understand. Unfortunately, I may not be of much help because I haven't used Python for development before. I can only provide some minor suggestions.
That said I believe your patch is the right way to fix this issue with the current hacky workaround for Abema, so feel free to open a PR with it.

Resolving patch:
diff --git a/yt_dlp/extractor/abematv.py b/yt_dlp/extractor/abematv.py
--- a/yt_dlp/extractor/abematv.py
+++ b/yt_dlp/extractor/abematv.py
@@ -12,7 +12,7 @@
 import urllib.request
 import urllib.response
 import uuid
-
+from ..utils.networking import clean_proxies
 from .common import InfoExtractor
 from ..aes import aes_ecb_decrypt
 from ..utils import (
@@ -35,7 +35,10 @@ def add_opener(ydl, handler):  # FIXME: Create proper API in .networking
     rh = ydl._request_director.handlers['Urllib']
     if 'abematv-license' in rh._SUPPORTED_URL_SCHEMES:
         return
-    opener = rh._get_instance(cookiejar=ydl.cookiejar, proxies=ydl.proxies)
+    headers = ydl.params['http_headers'].copy()
+    proxies = ydl.proxies.copy()
+    clean_proxies(proxies, headers)
+    opener = rh._get_instance(cookiejar=ydl.cookiejar, proxies=proxies)
     assert isinstance(opener, urllib.request.OpenerDirector)
     opener.add_handler(handler)
     rh._SUPPORTED_URL_SCHEMES = (*rh._SUPPORTED_URL_SCHEMES, 'abematv-license')


Test patch: