Repository: sopel-irc/sopel

Issue:
URLs with ending commas get included when Sopel v6.5.x parse them.
```
Examples:

[02:46pm] <Ant> http://www.tv.com/shows/family-guy/da-boom-25450/,
02:46PM <Crushinator> [ Not found - TV.com ] - www.tv.com
02:47PM <URL> [ Not found - TV.com ] - www.tv.com

Periods had no problems:
[02:48pm] <Ant> http://www.tv.com/shows/family-guy/da-boom-25450/.
02:48PM <URL> [ Family Guy - Season 2, Episode 3: Da Boom - TV.com ] -
          www.tv.com
02:48PM <Crushinator> [ Family Guy - Season 2, Episode 3: Da Boom - TV.com ] -
          www.tv.com

```


Discussion:
In fairness, it looks like the `URL` bot also has trouble parsing the link correctly. But this is a good chance for Sopel to be "better than the competition", I suppose. (Or are both bots Sopel-based?)

I'll play with making the URL matching in `url.py` a bit more robust at some point. Anyone else who feels like it is free to do the same. If we end up with multiple PRs addressing the issue, that just means we can pick the best approach.

FWIW, Sopel is pulling in the `.` as well. It's just that TV.com redirects to the canonical URL when a trailing dot is present.
Playing with how to do this, got a patch that works for these examples so far:

````
08:31:34 <~dgw>  https://www.google.com/,
08:31:34 <Sopel>  [ Google ] - www.google.com
08:31:51 <~dgw>  http://www.google.com/!
08:31:51 <Sopel>  [ Google ] - www.google.com
08:32:25 <~dgw>  http://www.google.com/; https://www.bing.com/; https://sopel.chat/
08:32:26 <Sopel>  [ Google ] - www.google.com
08:32:26 <Sopel>  [ Bing ] - www.bing.com
08:32:27 <Sopel>  [ Sopel - Sopel - The Python IRC Bot ] - sopel.chat
````

Want to put in some more time and get trailing closing brackets/parens handled right, too, before PR. :)
Thanks dgw. Please kindly let us know when it is up for us to try it.
The only issue I see with this is that commas are valid URL chars. So even though its not common, removing it is just a heuristic. I think the module making the request (web?) should handle this in one of two ways:
1. Make requests with trailing commas (and other valid punctuation), and on http error, try again without the punctuation. if it works, update the url for posterity to the correct comma-less version
2. Make requests without trailing commas, and on http error, try again with the punctuation. if it works, update the url in memory.

Option 2 may be better since more often that not, a comma is not meant to be included.

Both ways require the original detected URL be stored until needed; although, something like `URL` will automatically do one of these things immediately when it goes to get the `title`.

... might not be worth it. Also, it would require much more than a quick regex patch (which will likely be good enough for the vast majority of cases). Thoughts?
Several punctuation characters are valid in URLs, yes. As you said, @HumorBaby, removing them is a heuristic. Sopel is never going to be 100% correct whether or not it includes trailing punctuation, but omitting it is the least effort for the highest success rate. Not many URLs actually end with a punctuation character; most often, the trailing punctuation will be from the sentence in which the URL is included.

And in case you were thinking of working on this, I have been already. Just looking to finish up excluding closing brackets when a matching opening bracket does not occur in the URL. I self-assigned the issue to make it clearer that I'm already in-progress. :)

Resolving patch:
diff --git a/sopel/modules/url.py b/sopel/modules/url.py
--- a/sopel/modules/url.py
+++ b/sopel/modules/url.py
@@ -84,12 +84,26 @@ def setup(bot):
     if not bot.memory.contains('last_seen_url'):
         bot.memory['last_seen_url'] = tools.SopelMemory()
 
-    def find_func(text):
+    def find_func(text, clean=False):
+        def trim_url(url):
+            # clean trailing sentence- or clause-ending punctuation
+            while url[-1] in '.,?!\'":;':
+                url = url[:-1]
+
+            # clean unmatched parentheses/braces/brackets
+            for (opener, closer) in [('(', ')'), ('[', ']'), ('{', '}'), ('<', '>')]:
+                if url[-1] is closer and url.count(opener) < url.count(closer):
+                    url = url[:-1]
+
+            return url
+
         re_url = r'(?u)((?<!%s)(?:http|https|ftp)(?::\/\/\S+))'\
             % (bot.config.url.exclusion_char)
         r = re.compile(re_url, re.IGNORECASE)
 
         urls = re.findall(r, text)
+        if clean:
+            urls = [trim_url(url) for url in urls]
         return urls
 
     find_urls = find_func


Test patch: