Repository: yt-dlp/yt-dlp
Pull request #7603 (2023-07-15 22:38:57)

[networking] Add basic RH Preference System
This adds a basic framework for a RequestHandler Preference system. This allows us to dynamically change the order of RequestHandlers tried in the director per request, depending on various inputs.

This is useful for when we have multiple handlers that support a request however one may be preferred over the other depending on some factors.

It is all based around returning an integer that is used to calculate the order of handlers.

The tricky thing is getting the returned integer preference to line up with other handlers. In the future we could add some abstraction to this with standardized values for different features. However, as a start this should be more than enough.


<details open><summary>Template</summary> <!-- OPEN is intentional -->

<!--

# PLEASE FOLLOW THE GUIDE BELOW

- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes `[ ]` relevant to your *pull request* (like [x])
- Use *Preview* tab to see how your *pull request* will actually look like

-->

### Before submitting a *pull request* make sure you have:
- [X] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [X] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [X] Checked the code with [flake8](https://pypi.python.org/pypi/flake8) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
- [X] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)

### What is the purpose of your *pull request*?
- [ ] Fix or improvement to an extractor (Make sure to add/update tests)
- [ ] New extractor ([Piracy websites will not be accepted](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy))
- [ ] Core bug fix/improvement
- [X] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))


<!-- Do NOT edit/remove anything below this! -->
</details><details><summary>Copilot Summary</summary>  

<!--
copilot:all
-->
### <samp>🤖 Generated by Copilot at 80f0f60</samp>

### Summary
📦🧪🚀

<!--
1.  📦 for importing the `Preference` class to the `yt_dlp.networking` package.
2.  🧪 for adding and modifying tests for the `Preference` feature.
3.  🚀 for introducing the `Preference` feature and adding support for using it in the `YoutubeDL` class.
-->
This pull request adds a new feature called `Preference` to the `yt_dlp.networking` package, which allows the network requests to be sorted by the suitability of the `RequestHandler`s. It also adds and modifies tests for this feature and integrates it with the `YoutubeDL` class.

> _`RequestDirector` sorts the handlers by `Preference`_
> _A new feature to boost the network performance_
> _But beware of the dark side of this enhancement_
> _It may unleash the wrath of the `RequestHandler`s_

### Walkthrough
*  Implement and use the `Preference` feature to sort the `RequestHandler`s by their preference for a given request ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78R38), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78L997-R1004), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78L1025-R1107), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-be34debf65f24855b0906bd941d13c28e89a5e2328b3580c7f0805cb08f9beddR4), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abL37-R76), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abR88), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abR101-R114), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abL76-R130), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-d3ba8be45cae8dd7889a71c3360c9e4ac1160de8a5f3443b6e4a656395267f9bL37-R37), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-d3ba8be45cae8dd7889a71c3360c9e4ac1160de8a5f3443b6e4a656395267f9bR689-R692))
  * Add the `Preference` class and the `register_preference` function to the `yt_dlp.networking.common` module ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abL37-R76))
    * Define the preference logic for a specific `RequestHandler` or for any `RequestHandler` using attributes and methods ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abL37-R76))
    * Register a `Preference` subclass to the `_PREFERENCES` list using the `register_preference` decorator ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abL37-R76))
  * Add the `preferences` attribute and the `add_preference` and `_get_handlers` methods to the `RequestDirector` class in the `yt_dlp.networking.common` module ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abR88), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abR101-R114))
    * Store and access the `Preference` instances that are added to the `RequestDirector` instance using the `preferences` attribute ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abR88), [link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abR101-R114))
    * Add a `Preference` instance to the `RequestDirector` instance using the `add_preference` method ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abR101-R114))
    * Return a list of `RequestHandler`s sorted by their preference for a request using the `_get_handlers` method ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abR101-R114))
  * Modify the `send` method of the `RequestDirector` class in the `yt_dlp.networking.common` module to use the `_get_handlers` method instead of the reverse order of registration ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-ead7fca4af365bb8c89c1a0257a027147d0533eb2e0dfb395d08a9a12e58d4abL76-R130))
  * Import the `_PREFERENCES` list from the `yt_dlp.networking.common` module to the `yt_dlp/YoutubeDL.py` file ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-d3ba8be45cae8dd7889a71c3360c9e4ac1160de8a5f3443b6e4a656395267f9bL37-R37))
  * Add a loop to the `__init__` method of the `YoutubeDL` class in the `yt_dlp/YoutubeDL.py` file to add an instance of each `Preference` subclass to the `RequestDirector` instance ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-d3ba8be45cae8dd7889a71c3360c9e4ac1160de8a5f3443b6e4a656395267f9bR689-R692))
  * Import the `Preference` class from the `yt_dlp.networking.common` module to the `test/test_networking.py` file ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78R38))
  * Add several new test functions and classes to the `test/test_networking.py` file to test the `Preference` feature and its compatibility with the `RequestDirector` and `RequestHandler` classes ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78L1025-R1107))
    * Test the `RequestDirector`'s ability to use different types of `Preference`s, such as fixed, dynamic, and default preferences, in the `test_preference` function ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78L1025-R1107))
    * Test the `Preference` class's attributes and methods, such as `_RH_KEY`, `_PREFERENCE`, and `_get_preference`, in the `TestPreference` class ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78L1025-R1107))
  * Modify the `test_unsupported_handlers` function in the `test/test_networking.py` file to test that the `RequestDirector` can prioritize the `RequestHandler`s that can handle the request over those that cannot using a subclass of `Preference` called `SupportedRHPReference` ([link](https://github.com/yt-dlp/yt-dlp/pull/7603/files?diff=unified&w=0#diff-e92498eb46ad8369b16aed0d65fa7c8d56c81b2d595f6510981c6cfb0c0f7e78L997-R1004))



</details>



Patch:
diff --git a/yt_dlp/YoutubeDL.py b/yt_dlp/YoutubeDL.py
index 87bca5bbe..666d89b46 100644
--- a/yt_dlp/YoutubeDL.py
+++ b/yt_dlp/YoutubeDL.py
@@ -34,7 +34,7 @@
 from .extractor.openload import PhantomJSwrapper
 from .minicurses import format_text
 from .networking import HEADRequest, Request, RequestDirector
-from .networking.common import _REQUEST_HANDLERS
+from .networking.common import _REQUEST_HANDLERS, _RH_PREFERENCES
 from .networking.exceptions import (
     HTTPError,
     NoSupportingHandlers,
@@ -683,7 +683,7 @@ def process_color_policy(stream):
         self.params['http_headers'] = HTTPHeaderDict(std_headers, self.params.get('http_headers'))
         self._load_cookies(self.params['http_headers'].get('Cookie'))  # compat
         self.params['http_headers'].pop('Cookie', None)
-        self._request_director = self.build_request_director(_REQUEST_HANDLERS.values())
+        self._request_director = self.build_request_director(_REQUEST_HANDLERS.values(), _RH_PREFERENCES)
 
         if auto_init and auto_init != 'no_verbose_header':
             self.print_debug_header()
@@ -4077,7 +4077,7 @@ def urlopen(self, req):
         except HTTPError as e:  # TODO: Remove in a future release
             raise _CompatHTTPError(e) from e
 
-    def build_request_director(self, handlers):
+    def build_request_director(self, handlers, preferences=None):
         logger = _YDLLogger(self)
         headers = self.params['http_headers'].copy()
         proxies = self.proxies.copy()
@@ -4106,6 +4106,7 @@ def build_request_director(self, handlers):
                     },
                 }),
             ))
+        director.preferences.update(preferences or [])
         return director
 
     def encode(self, s):
diff --git a/yt_dlp/networking/common.py b/yt_dlp/networking/common.py
index 8fba8c1c5..584c7bb4d 100644
--- a/yt_dlp/networking/common.py
+++ b/yt_dlp/networking/common.py
@@ -31,8 +31,19 @@
 )
 from ..utils.networking import HTTPHeaderDict, normalize_url
 
-if typing.TYPE_CHECKING:
-    RequestData = bytes | Iterable[bytes] | typing.IO | None
+
+def register_preference(*handlers: type[RequestHandler]):
+    assert all(issubclass(handler, RequestHandler) for handler in handlers)
+
+    def outer(preference: Preference):
+        @functools.wraps(preference)
+        def inner(handler, *args, **kwargs):
+            if not handlers or isinstance(handler, handlers):
+                return preference(handler, *args, **kwargs)
+            return 0
+        _RH_PREFERENCES.add(inner)
+        return inner
+    return outer
 
 
 class RequestDirector:
@@ -40,12 +51,17 @@ class RequestDirector:
 
     Helper class that, when given a request, forward it to a RequestHandler that supports it.
 
+    Preference functions in the form of func(handler, request) -> int
+    can be registered into the `preferences` set. These are used to sort handlers
+    in order of preference.
+
     @param logger: Logger instance.
     @param verbose: Print debug request information to stdout.
     """
 
     def __init__(self, logger, verbose=False):
         self.handlers: dict[str, RequestHandler] = {}
+        self.preferences: set[Preference] = set()
         self.logger = logger  # TODO(Grub4k): default logger
         self.verbose = verbose
 
@@ -58,6 +74,16 @@ def add_handler(self, handler: RequestHandler):
         assert isinstance(handler, RequestHandler), 'handler must be a RequestHandler'
         self.handlers[handler.RH_KEY] = handler
 
+    def _get_handlers(self, request: Request) -> list[RequestHandler]:
+        """Sorts handlers by preference, given a request"""
+        preferences = {
+            rh: sum(pref(rh, request) for pref in self.preferences)
+            for rh in self.handlers.values()
+        }
+        self._print_verbose('Handler preferences for this request: %s' % ', '.join(
+            f'{rh.RH_NAME}={pref}' for rh, pref in preferences.items()))
+        return sorted(self.handlers.values(), key=preferences.get, reverse=True)
+
     def _print_verbose(self, msg):
         if self.verbose:
             self.logger.stdout(f'director: {msg}')
@@ -73,8 +99,7 @@ def send(self, request: Request) -> Response:
 
         unexpected_errors = []
         unsupported_errors = []
-        # TODO (future): add a per-request preference system
-        for handler in reversed(list(self.handlers.values())):
+        for handler in self._get_handlers(request):
             self._print_verbose(f'Checking if "{handler.RH_NAME}" supports this request.')
             try:
                 handler.validate(request)
@@ -530,3 +555,10 @@ def info(self):
     def getheader(self, name, default=None):
         deprecation_warning('Response.getheader() is deprecated, use Response.get_header', stacklevel=2)
         return self.get_header(name, default)
+
+
+if typing.TYPE_CHECKING:
+    RequestData = bytes | Iterable[bytes] | typing.IO | None
+    Preference = typing.Callable[[RequestHandler, Request], int]
+
+_RH_PREFERENCES: set[Preference] = set()


Test patch:
diff --git a/test/test_networking.py b/test/test_networking.py
index 9c33b0d4c..2622d24da 100644
--- a/test/test_networking.py
+++ b/test/test_networking.py
@@ -1035,17 +1035,17 @@ def test_send(self):
         assert isinstance(director.send(Request('http://')), FakeResponse)
 
     def test_unsupported_handlers(self):
-        director = RequestDirector(logger=FakeLogger())
-        director.add_handler(FakeRH(logger=FakeLogger()))
-
         class SupportedRH(RequestHandler):
             _SUPPORTED_URL_SCHEMES = ['http']
 
             def _send(self, request: Request):
                 return Response(fp=io.BytesIO(b'supported'), headers={}, url=request.url)
 
-        # This handler should by default take preference over FakeRH
+        director = RequestDirector(logger=FakeLogger())
         director.add_handler(SupportedRH(logger=FakeLogger()))
+        director.add_handler(FakeRH(logger=FakeLogger()))
+
+        # First should take preference
         assert director.send(Request('http://')).read() == b'supported'
         assert director.send(Request('any://')).read() == b''
 
@@ -1072,6 +1072,27 @@ def _send(self, request: Request):
         director.add_handler(UnexpectedRH(logger=FakeLogger))
         assert director.send(Request('any://'))
 
+    def test_preference(self):
+        director = RequestDirector(logger=FakeLogger())
+        director.add_handler(FakeRH(logger=FakeLogger()))
+
+        class SomeRH(RequestHandler):
+            _SUPPORTED_URL_SCHEMES = ['http']
+
+            def _send(self, request: Request):
+                return Response(fp=io.BytesIO(b'supported'), headers={}, url=request.url)
+
+        def some_preference(rh, request):
+            return (0 if not isinstance(rh, SomeRH)
+                    else 100 if 'prefer' in request.headers
+                    else -1)
+
+        director.add_handler(SomeRH(logger=FakeLogger()))
+        director.preferences.add(some_preference)
+
+        assert director.send(Request('http://')).read() == b''
+        assert director.send(Request('http://', headers={'prefer': '1'})).read() == b'supported'
+
 
 # XXX: do we want to move this to test_YoutubeDL.py?
 class TestYoutubeDLNetworking: