Repository: sopel-irc/sopel

Issue:
wikipedia: query parameters cause URL parse failure
### Description

Wikipedia URLs that contain `?query=params` can confuse Sopel, leading to a `KeyError`:

```
10:31 <+SnoopJ> Sopel, do you beef it on this URL? https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form
10:31 <+Sopel> Unexpected KeyError ('parse') from SnoopJ at 2023-02-17 15:31:51.723938. Message was: https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form
```

### Reproduction steps

1. Send a Wikipedia URL with an embedded query parameter to a Sopel instance with the `wikipedia` plugin active

### Expected behavior

The same article-retrieval behavior as a Wikipedia URL without query parameters:

```
10:51 <+SnoopJ> Sopel, https://en.wikipedia.org/wiki/Gauss%27s_law#Differential_form
10:51 <+Sopel> [wikipedia] Gauss's law - Differential form | "By the divergence theorem, Gauss's law can alternatively be written in the differential form: ∇ ⋅ E = 
ρ ε 0 ε r {\displaystyle \nabla \cdot \mathbf {E} ={\frac {\rho }{\varepsilon _{0}\varepsilon _{r}}}} where ∇ · E is the divergence of the electric field, ε0 is the vacuum permittivity, ε r {\displaystyle \varepsilon _{r}} is the relative permittivity, and ρ is the volume charge density […]"
```

### Relevant logs

_No response_

### Notes

Exception details from the error log:

```
[2023-02-17 11:16:12,662] sopel.bot            ERROR    - Unexpected error ('parse') from SnoopJ at 2023-02-17 16:16:12.662537. Message was: https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form                                                                                                                   
Traceback (most recent call last):                                                                                                                                    
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/bot.py", line 763, in call_rule                                                                 
    rule.execute(sopel, trigger)                                                                                                                                      
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/plugins/rules.py", line 1057, in execute                                                        
    exit_code = self._handler(bot, trigger)                                                                                                                           
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/plugins/rules.py", line 1608, in execute_handler                                                
    return handler(bot, trigger, match=trigger)                                                                                                                       
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/modules/wikipedia.py", line 300, in mw_info                                                     
    say_section(bot, trigger, match.group(1), unquote(match.group(2)), unquote(match.group(3)))                                                                       
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/modules/wikipedia.py", line 243, in say_section                                                 
    snippet = mw_section(server, query, section)                                                                                                                      
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/modules/wikipedia.py", line 264, in mw_section                                                  
    for entry in sections['parse']['sections']:                                                                                                                       
KeyError: 'parse'
```

### Sopel version

7.1.9 (also happens on `master`)

### Installation method

`pip install`

### Python version

3.9.9

### Operating system

Ubuntu 20.04

### IRCd

_No response_

### Relevant plugins

wikipedia
wikipedia: query parameters cause URL parse failure
### Description

Wikipedia URLs that contain `?query=params` can confuse Sopel, leading to a `KeyError`:

```
10:31 <+SnoopJ> Sopel, do you beef it on this URL? https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form
10:31 <+Sopel> Unexpected KeyError ('parse') from SnoopJ at 2023-02-17 15:31:51.723938. Message was: https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form
```

### Reproduction steps

1. Send a Wikipedia URL with an embedded query parameter to a Sopel instance with the `wikipedia` plugin active

### Expected behavior

The same article-retrieval behavior as a Wikipedia URL without query parameters:

```
10:51 <+SnoopJ> Sopel, https://en.wikipedia.org/wiki/Gauss%27s_law#Differential_form
10:51 <+Sopel> [wikipedia] Gauss's law - Differential form | "By the divergence theorem, Gauss's law can alternatively be written in the differential form: ∇ ⋅ E = 
ρ ε 0 ε r {\displaystyle \nabla \cdot \mathbf {E} ={\frac {\rho }{\varepsilon _{0}\varepsilon _{r}}}} where ∇ · E is the divergence of the electric field, ε0 is the vacuum permittivity, ε r {\displaystyle \varepsilon _{r}} is the relative permittivity, and ρ is the volume charge density […]"
```

### Relevant logs

_No response_

### Notes

Exception details from the error log:

```
[2023-02-17 11:16:12,662] sopel.bot            ERROR    - Unexpected error ('parse') from SnoopJ at 2023-02-17 16:16:12.662537. Message was: https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form                                                                                                                   
Traceback (most recent call last):                                                                                                                                    
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/bot.py", line 763, in call_rule                                                                 
    rule.execute(sopel, trigger)                                                                                                                                      
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/plugins/rules.py", line 1057, in execute                                                        
    exit_code = self._handler(bot, trigger)                                                                                                                           
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/plugins/rules.py", line 1608, in execute_handler                                                
    return handler(bot, trigger, match=trigger)                                                                                                                       
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/modules/wikipedia.py", line 300, in mw_info                                                     
    say_section(bot, trigger, match.group(1), unquote(match.group(2)), unquote(match.group(3)))                                                                       
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/modules/wikipedia.py", line 243, in say_section                                                 
    snippet = mw_section(server, query, section)                                                                                                                      
  File "/home/snoopjedi/.sopel/venv/lib/python3.9/site-packages/sopel/modules/wikipedia.py", line 264, in mw_section                                                  
    for entry in sections['parse']['sections']:                                                                                                                       
KeyError: 'parse'
```

### Sopel version

7.1.9 (also happens on `master`)

### Installation method

`pip install`

### Python version

3.9.9

### Operating system

Ubuntu 20.04

### IRCd

_No response_

### Relevant plugins

wikipedia


Discussion:
Note that this seems to happen for any query parameters, not just the key/val pair I noticed:

```
10:54 <SnoopJ> https://en.wikipedia.org/wiki/Gauss%27s_law?someparam=someval#Differential_form
10:54 <terribot> Unexpected error ('parse') from SnoopJ at 2023-02-17 15:54:59.015063. Message was: https://en.wikipedia.org/wiki/Gauss%27s_law?someparam=someval#Differential_form
```

This _does not_ seem to occur if there is no anchor, so this may be an interaction between query parameters and anchors

```
10:54 <SnoopJ> https://en.wikipedia.org/wiki/Gauss%27s_law?someparam=someval
[no Sopel response]
```
On the other hand:

```python
>>> import re
>>> p = re.compile(r'https?:\/\/([a-z]+(?:\.m)?\.wikipedia\.org)\/wiki\/((?!File\:)[^ #]+)#?([^ ]*)')
>>> m = p.match('https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form')
>>> m.groups()
('en.wikipedia.org', 'Gauss%27s_law?useskin=vector', 'Differential_form')
```

This is the regex used in the plugin, you can see that the second group is now `Gauss%27s_law?useskin=vector` so maybe all that is needed is to remove the part after `?`.

If I remove the anchor part, here is the result:

```python
>>> m2 = p.match('https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector')
>>> m2.groups()
('en.wikipedia.org', 'Gauss%27s_law?useskin=vector', '')
```
One option is to take only the part of the page title before any `?` character (if it appears _in_ the page title, it will be escaped to `%3F`, e.g. [this species article](https://en.wikipedia.org/wiki/%3FOryzomys_pliocaenicus)).

The other option is to modify the regex such that the title portion matches `[^ #?]+` with an optional `(?:\?[^ #]+)?` group between it and the fragment. I would prefer this one for code-smell reasons, but it's simultaneously the technically cleaner way _and_ the more complicated method.
To be honest, the best way is probably to stop doing regex for that and use the good old `urllib.parse.urlparse`:

```python
>>> from urllib.parse import urlparse
>>> urlparse('https://en.wikipedia.org/wiki/%3FOryzomys_pliocaenicus')
ParseResult(scheme='https', netloc='en.wikipedia.org', path='/wiki/%3FOryzomys_pliocaenicus', params='', query='', fragment='')
>>> urlparse('https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector')
ParseResult(scheme='https', netloc='en.wikipedia.org', path='/wiki/Gauss%27s_law', params='', query='useskin=vector', fragment='')
>>> urlparse('https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form')
ParseResult(scheme='https', netloc='en.wikipedia.org', path='/wiki/Gauss%27s_law', params='', query='useskin=vector', fragment='Differential_form')
```

And then use a regex to work on the path only. This way, we would not have to deal with using a regex to detect the anchor, the query parameter, or what have you, and only work on the `path` of the URL to remove the prefix (such as `/wiki`).
You know what? You're right. Our plugins (built-in and external) _do not_ take enough advantage of `urllib.parse`.

Pattern for Wikipedia article links can be something simple, like just the first part that excludes `File:` URLs, and lean on `urlparse` for the rest. 👍
Note that this seems to happen for any query parameters, not just the key/val pair I noticed:

```
10:54 <SnoopJ> https://en.wikipedia.org/wiki/Gauss%27s_law?someparam=someval#Differential_form
10:54 <terribot> Unexpected error ('parse') from SnoopJ at 2023-02-17 15:54:59.015063. Message was: https://en.wikipedia.org/wiki/Gauss%27s_law?someparam=someval#Differential_form
```

This _does not_ seem to occur if there is no anchor, so this may be an interaction between query parameters and anchors

```
10:54 <SnoopJ> https://en.wikipedia.org/wiki/Gauss%27s_law?someparam=someval
[no Sopel response]
```
On the other hand:

```python
>>> import re
>>> p = re.compile(r'https?:\/\/([a-z]+(?:\.m)?\.wikipedia\.org)\/wiki\/((?!File\:)[^ #]+)#?([^ ]*)')
>>> m = p.match('https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form')
>>> m.groups()
('en.wikipedia.org', 'Gauss%27s_law?useskin=vector', 'Differential_form')
```

This is the regex used in the plugin, you can see that the second group is now `Gauss%27s_law?useskin=vector` so maybe all that is needed is to remove the part after `?`.

If I remove the anchor part, here is the result:

```python
>>> m2 = p.match('https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector')
>>> m2.groups()
('en.wikipedia.org', 'Gauss%27s_law?useskin=vector', '')
```
One option is to take only the part of the page title before any `?` character (if it appears _in_ the page title, it will be escaped to `%3F`, e.g. [this species article](https://en.wikipedia.org/wiki/%3FOryzomys_pliocaenicus)).

The other option is to modify the regex such that the title portion matches `[^ #?]+` with an optional `(?:\?[^ #]+)?` group between it and the fragment. I would prefer this one for code-smell reasons, but it's simultaneously the technically cleaner way _and_ the more complicated method.
To be honest, the best way is probably to stop doing regex for that and use the good old `urllib.parse.urlparse`:

```python
>>> from urllib.parse import urlparse
>>> urlparse('https://en.wikipedia.org/wiki/%3FOryzomys_pliocaenicus')
ParseResult(scheme='https', netloc='en.wikipedia.org', path='/wiki/%3FOryzomys_pliocaenicus', params='', query='', fragment='')
>>> urlparse('https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector')
ParseResult(scheme='https', netloc='en.wikipedia.org', path='/wiki/Gauss%27s_law', params='', query='useskin=vector', fragment='')
>>> urlparse('https://en.wikipedia.org/wiki/Gauss%27s_law?useskin=vector#Differential_form')
ParseResult(scheme='https', netloc='en.wikipedia.org', path='/wiki/Gauss%27s_law', params='', query='useskin=vector', fragment='Differential_form')
```

And then use a regex to work on the path only. This way, we would not have to deal with using a regex to detect the anchor, the query parameter, or what have you, and only work on the `path` of the URL to remove the prefix (such as `/wiki`).
You know what? You're right. Our plugins (built-in and external) _do not_ take enough advantage of `urllib.parse`.

Pattern for Wikipedia article links can be something simple, like just the first part that excludes `File:` URLs, and lean on `urlparse` for the rest. 👍

Resolving patch:
diff --git a/sopel/modules/wikipedia.py b/sopel/modules/wikipedia.py
--- a/sopel/modules/wikipedia.py
+++ b/sopel/modules/wikipedia.py
@@ -10,12 +10,12 @@
 from html.parser import HTMLParser
 import logging
 import re
+from urllib.parse import quote, unquote, urlparse
 
 from requests import get
 
 from sopel import plugin
 from sopel.config import types
-from sopel.tools.web import quote, unquote
 
 
 LOGGER = logging.getLogger(__name__)
@@ -321,26 +321,27 @@ def mw_image_description(server, image):
     return desc
 
 
-# Matches a wikipedia page (excluding spaces and #, but not /File: links), with a separate optional field for the section
-@plugin.url(r'https?:\/\/([a-z]+(?:\.m)?\.wikipedia\.org)\/wiki\/((?!File\:)[^ #]+)#?([^ ]*)')
+# Matches a Wikipedia link (excluding /File: pages)
+@plugin.url(r'https?:\/\/([a-z]+(?:\.m)?\.wikipedia\.org)\/wiki\/((?!File\:)[^ ]+)')
 @plugin.output_prefix(PLUGIN_OUTPUT_PREFIX)
 def mw_info(bot, trigger, match=None):
     """Retrieves and outputs a snippet of the linked page."""
     server = match.group(1)
-    query = unquote(match.group(2))
-    section = unquote(match.group(3))
+    page_info = urlparse(match.group(2))
+    article = unquote(page_info.path)
+    section = unquote(page_info.fragment)
 
     if section:
         if section.startswith('cite_note-'):  # Don't bother trying to retrieve a snippet when cite-note is linked
-            say_snippet(bot, trigger, server, query, show_url=False)
+            say_snippet(bot, trigger, server, article, show_url=False)
         elif section.startswith('/media'):
             # gh2316: media fragments are usually images; try to get an image description
             image = section[7:]  # strip '/media' prefix in pre-3.9 friendly way
             say_image_description(bot, trigger, server, image)
         else:
-            say_section(bot, trigger, server, query, section)
+            say_section(bot, trigger, server, article, section)
     else:
-        say_snippet(bot, trigger, server, query, show_url=False)
+        say_snippet(bot, trigger, server, article, show_url=False)
 
 
 @plugin.command('wikipedia', 'wp')


Test patch: