Repository: sopel-irc/sopel

Issue:
url: documented as calling other plugins when redirects match, but doesn't
Once upon a time, the `url` plugin's `title_auto` feature would check if a link _redirected to_ a URL that should be handled by another plugin, but that appears to no longer be the case since [the conversion to use `requests`](https://github.com/sopel-irc/sopel/commit/ccda12e3f5440ee87edc2689faccca1f3f027f4d). Its documentation and that for the `process_urls` helper (below) still say this is the case, but the behavior is gone.

https://github.com/sopel-irc/sopel/blob/d36a19dad548068e34357d63761258cdd5e5efc0/sopel/modules/url.py#L290-L295
https://github.com/sopel-irc/sopel/blob/d36a19dad548068e34357d63761258cdd5e5efc0/sopel/modules/url.py#L322-L329

I would argue that this is a useful feature, and we should bring it back. For example, our first-party GitHub plugin could get free handling of repo/issue/PR/commit links shortened through git.io if this was still the case. _Any_ shortened URL at present, actually, is only ever going to get the generic title-fetching behavior.

While it's certainly possible to create Yet Another Sopel Plugin that resolves short URLs, it would be fairly annoying to implement—and it wouldn't block the built-in title handling, either.

_Note, I don't care if this is part of `url`, but it makes more sense to put it there than into the bot's core URL handling. We don't necessarily want the core bot making HTTP requests every time it sees a link. Some of the thoughts in this issue were previously expressed in #sopel@libera on 2021-11-30._


Discussion:
I believe the _"did any redirects happen?"_ part of the fix for this issue involves looking at [`Response.history`](https://requests.readthedocs.io/en/latest/api/#requests.Response.history), but I'm less sure about what to do after that after a glance at the plugin to remind myself of its structure.

A patch may want to add a note to `plugin.url` to explicitly warn users that they will _not_ receive events for redirects to the URLs they declare this way (since no request is performed)
> A patch may want to add a note to `plugin.url` to explicitly warn users that they will _not_ receive events for redirects to the URLs they declare this way (since no request is performed)

That would constitute an inability to re-implement the documented behavior, then, wouldn't it?

> "If that redirected URL should be handled by another plugin, dispatch the callback for it."
The URL plugin should probably say to request not to follow redirect, and to implement the follow-redirect logic ourselves. This way, we should be able to look at a request, if it's a redirect, ask the bot if a plugin can deal with it, etc. and so on up until there is no redirect and no plugin can handle it.
Or the bot's URL dispatcher can test for redirects itself if no plugin's handler matches? I wouldn't mind decoupling this redirect-handling behavior from `url` at all. It would be ideal, in fact.

Problem is the `url` plugin's `auto_title` handler also counts as "a plugin's handler" that matches everything. But I'd also say that it's better to set the _URL dispatcher_ to ignore `url.auto_title` than to rely on `url` to handle redirected links. It's still coupling core behavior to a plugin, but much less obnoxious than the other way around.
If we're considering that, my dream is something like:
- Sopel detects URL
- If a plugin configured a callback for that URL, hand it to the plugin and you're done. Otherwise,
- Sopel performs GET request. If redirect, goto ^
- Sopel passes the URL *and the `Response`* to plugins that have registered for page body processing
  - Plugins may be able to register an xpath or callback for deciding if they're relevant
  - Callbacks can also return E_NOTME or something, which tells Sopel to continue looking for other plugins to handle that page
- If all registered plugins pass on it (url, xpath, or E_NOTME), it goes to url.py if enabled

This would let plugins like bugzilla or a hypothetical mastodon/activitypub plugin to handle the relevant application regardless of the domain/path it lives at (mastodon.social, nondeterministic.computer, etc etc etc etc etc), and for others that use page data instead of turning a URL into an API call, they could dump it directly through an xml parser or BS4.

Worth it? No idea.
We've talked about that idea before, and I like it. Not this late in 8.0, and it's big enough I think it has to wait for 9.0, but I like it. If you're up for taking your bullets there and fleshing them out a bit into a standalone feature issue, go for it. 😃

FWIW, a mastodon plugin is [no longer hypothetical](https://github.com/SnoopJ/sopel-mastodon) (soon to be adopted under our org, I think).

The question now is, how can we resolve _this_ issue in a relatively low-effort way for 8.0, while planning ahead to replace it in 9?
**(semantic warning: longposting)**

> That would constitute an inability to re-implement the documented behavior, then, wouldn't it?

I think I'm keying mostly off of your suggestion that the patch that implements this should live in `url`, which I understood as meaning that _this plugin_ should be capable of dealing with redirects, but I think that the answer to your question is _"yes"_ the way I've been thinking about it. I agree with your later reply that the general problem does not make sense for 8.0, it may be enough to get `url` alone dealing with redirections, and we might learn more about how we want to solve the big problem in pursuing that.

### Sopel doing requests on the plugins behalf

>     * Sopel performs GET request. If redirect, goto ^

I'm a -1 on Sopel doing eager requests, I don't think the core should be performing a request on every matched URL _before_ dispatching to plugins. What should the rules for this one-size-fits all request look like? What if someone links to an `.iso` etc? There's a lot to worry about, and I don't really think it's the core's business as a general rule. At a minimum, we seem to be talking about paying attention to _every_ URL, which is a pretty big change.

_However_, reflecting on this suggestion, I began wondering if it makes sense for Sopel to provide a helper for requests that _will_ dispatch to plugins that have a registered pattern matching an eventual redirect. Let's assume `https://example.com` redirects to some other specific path, and that two plugins have registered patterns for this domain. Does it make sense to do something like this?

```python
@plugin.url("https://example.com")
def pluginA_example(bot, trigger):
    urls = web.search_urls(trigger, exclusion_char=bot.config.url.exclusion_char, clean=True)
    for url in urls:
        # in the servicing of get(), we could dispatch to decorators that match the redirect URL, where appropriate
        response = web.get(url)  # type: requests.Response

    # ...

# …somewhere else, a cousin decoration occurs…
@plugin.url("https://example.com/with/a/specific/path")
def pluginB_example_specific(bot, trigger):
    # ...
```

Other spellings of this _"let the bot do the request"_ ideas might include a helper for explicitly delegating back to the bot (e.g. `bot.url_event(redirect_target)`), or some way for plugin authors to explicitly opt in to letting the core do the request for them (e.g. `@plugin.url("https://example.com/", do_request_for_me=True)`). I think these options are along the same lines as what @Exirel suggested.

I think all of that is definitely well past the bar for 8.0, but it's interesting to think about how to get redirects working cooperatively.
I've filed #2432 as longer-term planning for restoration of this feature, so that for 8.0 we can focus on 'just' fixing the incorrect claim in the docstrings. It's a good idea to dispatch to callbacks who want to know about redirects, but I think we need to think about that some more.

Resolving patch:
diff --git a/sopel/modules/url.py b/sopel/modules/url.py
--- a/sopel/modules/url.py
+++ b/sopel/modules/url.py
@@ -13,7 +13,7 @@
 from ipaddress import ip_address
 import logging
 import re
-from typing import Generator, List, Optional, Tuple, TYPE_CHECKING
+from typing import Generator, List, NamedTuple, Optional, TYPE_CHECKING
 from urllib.parse import urlparse
 
 import dns.resolver
@@ -289,10 +289,10 @@ def title_command(bot: SopelWrapper, trigger: Trigger):
         # needs to be a list so len() can be checked later
         urls = list(web.search_urls(trigger))
 
-    for url, title, domain, tinyurl, dispatched in process_urls(
+    for url, title, domain, tinyurl, ignored in process_urls(
         bot, trigger, urls, requested=True
     ):
-        if dispatched:
+        if ignored:
             result_count += 1
             continue
         message = "%s | %s" % (title, domain)
@@ -317,8 +317,13 @@ def title_command(bot: SopelWrapper, trigger: Trigger):
 def title_auto(bot: SopelWrapper, trigger: Trigger):
     """
     Automatically show titles for URLs. For shortened URLs/redirects, find
-    where the URL redirects to and show the title for that (or call a function
-    from another plugin to give more information).
+    where the URL redirects to and show the title for that.
+
+    .. note::
+
+        URLs that match (before redirection) any other registered callbacks
+        will *not* have their titles shown.
+
     """
     # Enabled or disabled by feature flag
     if not bot.settings.url.enable_auto_title:
@@ -342,8 +347,8 @@ def title_auto(bot: SopelWrapper, trigger: Trigger):
             continue
         urls.append(url)
 
-    for url, title, domain, tinyurl, dispatched in process_urls(bot, trigger, urls):
-        if not dispatched:
+    for url, title, domain, tinyurl, ignored in process_urls(bot, trigger, urls):
+        if not ignored:
             message = '%s | %s' % (title, domain)
             if tinyurl:
                 message += ' ( %s )' % tinyurl
@@ -353,16 +358,30 @@ def title_auto(bot: SopelWrapper, trigger: Trigger):
         bot.memory["last_seen_url"][trigger.sender] = url
 
 
+class URLInfo(NamedTuple):
+    """Helper class for information about a URL handled by this plugin."""
+
+    url: str
+
+    title: Optional[str]
+    """The title associated with ``url``, if appropriate."""
+
+    hostname: Optional[str]
+    """The hostname associated with ``url``, if appropriate."""
+
+    tinyurl: Optional[str]
+    """A shortened form of ``url``, if appropriate."""
+
+    ignored: bool
+    """Whether or not this URL matches any registered callbacks or is explicitly excluded."""
+
+
 def process_urls(
     bot: SopelWrapper,
     trigger: Trigger,
     urls: List[str],
     requested: bool = False,
-) -> Generator[
-    Tuple[str, Optional[str], Optional[str], Optional[str], bool],
-    None,
-    None,
-]:
+) -> Generator[URLInfo, None, None]:
     """
     For each URL in the list, ensure it should be titled, and do so.
 
@@ -371,15 +390,25 @@ def process_urls(
     :param urls: The URLs detected in the triggering message
     :param requested: Whether the title was explicitly requested (vs automatic)
 
-    See if it's handled by another plugin. If not, find where it redirects to,
-    if anywhere. If that redirected URL should be handled by another plugin,
-    dispatch the callback for it. Return a list of
-    (url, title, hostname, tinyurl, dispatched) tuples for each URL.
+    Yields a tuple ``(url, title, hostname, tinyurl, ignored)`` for each URL.
+
+    .. note:
+
+        If a URL in ``urls`` has any registered callbacks, this function will NOT
+        retrieve the title, and considers the URL as dispatched to those callbacks.
+        In this case, only the ``url`` and ``ignored=True`` will be set; all
+        other values will be ``None``.
+
+    .. note:
 
-    If a callback was dispatched, only the url and dispatched=True will be set.
+        For titles explicitly requested by the user, ``exclusion_char`` and
+        exclusions from the ``.urlban``/``.urlpban`` commands are skipped.
+
+    .. versionchanged:: 8.0
+
+        This function **does not** notify callbacks registered for URLs
+        redirected to from URLs passed to this function. See #2432, #2230.
 
-    For titles explicitly requested by the user, exclusion_char and excludes
-    are skipped.
     """
     shorten_url_length = bot.config.url.shorten_url_length
     for url in urls:
@@ -389,10 +418,10 @@ def process_urls(
 
         parsed_url = urlparse(url)
 
-        # Check the URL does not match an existing URL callback
         if check_callbacks(bot, url, use_excludes=not requested):
-            yield (url, None, None, None, True)
-            return
+            # URL matches a callback OR is excluded, ignore
+            yield URLInfo(url, None, None, None, True)
+            continue
 
         # Prevent private addresses from being queried if enable_private_resolution is False
         # FIXME: This does nothing when an attacker knows how to host a 302


Test patch: