Repository: sopel-irc/sopel

Issue:
wikipedia: Message boxes are included in snippet
As of this writing, the issue is easily reproduced by sending a link to https://en.wikipedia.org/wiki/Gurmukhi#Letters in a channel where Sopel and the `wikipedia` plugin will see it:

```
<Sopel> [wikipedia] Gurmukhi - Letters | "This article contains IPA phonetic symbols. Without proper rendering
        support, you may see question marks, boxes, or other symbols instead of Unicode characters. For an
        introductory guide on IPA symbols, see Help:IPA. The Gurmukhī alphabet contains thirty-five base letters
        (akkhara, plural akkharā̃), traditionally arranged in seven rows of five letters each. The first three
        letters, or mātarā vāhak ("vowel […]"
```

The leading text there ("This article contains IPA" through "see Help:IPA.") comes from a message box template. Wikipedia (the English version, at least) has many such templates for IPA, various non-Roman scripts, and numerous categories of "issues" with articles such as insufficient sources. They're good for informing users about potential problems with rendering, or pointing out ways that the reader could help improve the page if they have the required knowledge. However, it's not useful to output them as part of the page snippet on IRC.

Preliminary research indicates that they generally use the [Message box](https://en.wikipedia.org/wiki/Module:Message_box) Lua module as a template base, and therefore filtering out `<table>`s with one of these seven class names (modified quote from [a snippet of Mbox docs](https://en.wikipedia.org/wiki/Template:Mbox_templates_see_also)) should skip most of them:

- `ambox`, for messageboxes on article pages.
- `cmbox`, for messageboxes on category pages.
- `imbox`, for messageboxes on file (image) pages.
- `tmbox`, for messageboxes on talk pages.
- `fmbox`, for header and footer messageboxes.
- `ombox`, for messageboxes on other types of page.
- `mbox`, for messageboxes that are used in different namespaces and change their presentation accordingly.

The plugin's parser logic already provides for skipping over irrelevant content; it shouldn't be too painful to add these boxes.


Discussion:

Resolving patch:
diff --git a/sopel/modules/wikipedia.py b/sopel/modules/wikipedia.py
--- a/sopel/modules/wikipedia.py
+++ b/sopel/modules/wikipedia.py
@@ -29,6 +29,7 @@ def __init__(self, section_name):
         self.section_name = section_name
 
         self.citations = False
+        self.messagebox = False
         self.span_depth = 0
         self.div_depth = 0
 
@@ -49,7 +50,9 @@ def handle_starttag(self, tag, attrs):
                     if attr[0] == 'class' and 'edit' in attr[1]:
                         self.span_depth += 1
 
-        elif tag == 'div':  # We want to skip thumbnail text and the inexplicable table of contents, and as such also need to track div depth
+        elif tag == 'div':
+            # We want to skip thumbnail text and the inexplicable table of contents,
+            # and as such also need to track div depth
             if self.div_depth:
                 self.div_depth += 1
             else:
@@ -57,6 +60,25 @@ def handle_starttag(self, tag, attrs):
                     if attr[0] == 'class' and ('thumb' in attr[1] or attr[1] == 'toc'):
                         self.div_depth += 1
 
+        elif tag == 'table':
+            # Message box templates are what we want to ignore here
+            for attr in attrs:
+                if (
+                    attr[0] == 'class'
+                    and any(classname in attr[1].lower() for classname in [
+                        # Most of list from https://en.wikipedia.org/wiki/Template:Mbox_templates_see_also
+                        'ambox',  # messageboxes on article pages
+                        'cmbox',  # messageboxes on category pages
+                        'imbox',  # messageboxes on file (image) pages
+                        'tmbox',  # messageboxes on talk pages
+                        'fmbox',  # header and footer messageboxes
+                        'ombox',  # messageboxes on other types of page
+                        'mbox',  # for messageboxes that are used in different namespaces and change their presentation accordingly
+                        'dmbox',  # for disambiguation messageboxes
+                    ])
+                ):
+                    self.messagebox = True
+
         elif tag == 'ol':
             for attr in attrs:
                 if attr[0] == 'class' and 'references' in attr[1]:
@@ -71,9 +93,11 @@ def handle_endtag(self, tag):
             self.span_depth -= 1
         if self.div_depth and tag == 'div':
             self.div_depth -= 1
+        if self.messagebox and tag == 'table':
+            self.messagebox = False
 
     def handle_data(self, data):
-        if self.consume and not any([self.citations, self.span_depth, self.div_depth]):
+        if self.consume and not any([self.citations, self.messagebox, self.span_depth, self.div_depth]):
             if not (self.is_header and data == self.section_name):  # Skip the initial header info only
                 self.result += data
 


Test patch: