Repository: sopel-irc/sopel

Issue:
help: pastebin provider should be configurable
The `help` module used to use GitHub's Gist function, but anonymous Gists were deprecated (see #1254).

We switched to ptpb.pw as a pastebin provider (see #1257) for Sopel 6.5.2, but for a short while yesterday (November 30, 2018) its domain was in an expired state. While the domain has been restored, the notice @kwaaak sent me on IRC about it being gone gave me a good scare.

I think it's prudent to start thinking about how to make `help.py` more configurable so a more permanent outage of ptpb.pw won't break Sopel's help functionality until the next release. Our release cadence is slow enough that it's a valid concern. I want to keep trying to minimize the number of things that can break in a way that gives the user no recourse but to wait (like `etymology.py`—see #1248). New versions should come out when they're done, and not be rushed because something is broken in the latest release.

In #1212, it was suggested that the use of a pastebin should be configurable, with the option to use the old behavior of just sending the command list to IRC. Also in that issue, I brought up the idea of dumping to a local file on Sopel's disk and linking to it, kind of like how `meetbot.py`'s meeting notes work.

On top of those features, I propose a new `ChoiceAttribute` with several different supported pastebin providers. `ptpb` would be one, of course.

**Suggestions for additional pastebin services are welcome in the comments!** Any service like `pastebin`, `hastebin`, etc. is fine as long as it seems unlikely to vanish in the foreseeable future.

----

Here is the current list of possibilities:

  * ~~`ptpb.pw`~~ shut down (ptpb/pb#246)
  * ~~`pastebin.com`~~ requires API key
  * `hastebin.com`
  * `0x0.st`
  * `termbin.com`
  * `clbin.com` — default as of v6.6.5


Discussion:
Hi!

From my understanding you're not working on this yet? Mind if I give it a go, along with related tickets? I'm looking to gain a bit of experience in Python. This looks like something I should be able to tackle.

Regardless of who does it, I spent a little time checking out the pastebin services you suggest. This might help.

## pastebin.com
This one, according to the API documentation, requires a dev key. We'd probably need to give the user a config option to provide a dev key. Seems like unnecessary hassle I think?

## hastebin.com
They don't seem to document their API which worries me (no claims of it being maintained the way it works now), but I figured out how stuff is processed there nonetheless. This seems to work:

```python
>>> r = requests.post('https://hastebin.com/documents', data = 'stuff')
>>> r.text
'{"key":"nepiwelixa"}'
```


## 0x0.st
Official API, no devkeys needed.

```python
>>> r = requests.post('http://0x0.st', files = {'file':'stuff\n'})
>>> r.text
'http://0x0.st/s7t0.txt\n'
```
> From my understanding you're not working on this yet? Mind if I give it a go, along with related tickets?

Certainly! Feel free to throw out ideas for other services, too; I just listed a few off the top of my head when writing this particular issue. I recently discovered `termbin.com`, for example, which is absolutely bare-bones and expires pastes after a month—no need to worry about stale help output getting left behind. I'll add it to the list of ideas in the OP.

As far as `pastebin.com` itself goes, you're right that the API key requirement is basically an immediate disqualifier for inclusion in Sopel's core. (Again, I made that list without researching APIs first. :sweat_smile:) Therefore, I'll remove it from the OP's list of ideas.
This is overkill, but if someone wants to flex their Python muscles, this is an opportunity: Use a PrivateBin pastebin service, https://github.com/PrivateBin/PrivateBin/wiki/PrivateBin-Directory
Definitely overkill. That said, I'd at least _consider_ a PR that added it as an option, but preferably it would come separately after the basic configuration setting is done.
I think termbin should be easy enough to add via a bare-bones TCP socket. Will look into it.

I've been toying with the idea of using software like `wgetpaste` (popular on Gentoo) or `pastebinit` (it's in Debian/Ubuntu repos), but the former is a bit niche and the latter seems outdated, at least on my system. These applications are CLI utilities that let you submit stuff to various pastebin services. Integrating one with Sopel would relieve us from having to maintain a list of working pastebins and the methods to use them. But alas.
> These applications are CLI utilities that let you submit stuff to various pastebin services. Integrating one with Sopel would relieve us from having to maintain a list of working pastebins and the methods to use them. But alas.

Alas, indeed. It would probably also relieve us of ever supporting Windows properly. Maybe even macOS.

It's not that those tools can't be run on non-Linux machines—they're just Bash & Python scripts, respectively. But if they're niche on Linux, imagine how hard it would be to get them installed on Windows! (Well, "hard" is not quite it, but it's probably not as simple as a package install command.) And that's assuming those scripts even support Windows themselves, which I haven't tested. It also seems kind of silly to spawn an external process to do something that Sopel could pretty easily do on its own.

One possible outcome of this feature idea, if someone _really_ wants to get down and dirty with the help code, is a truly generic pastebin system that can support (nearly) any site. Past, present, or future. I imagine it's doable with just a few options:

* `pastebin_api` — the endpoint URL; APIs that require a key (like `pastebin.com`) should just have the user include it here instead of adding a separate setting and then having to do string formatting
* `pastebin_request_type` — `POST` (the most common, and default if unset) `GET`
* `pastebin_text_field` — the request field in which the text should be sent (commonly `file` or `text`)
* `pastebin_options` — might be useful to pass extra options as `key=value` pairs, like to set paste expiration or something

_(Field names are for illustration and not necessarily the best choices, and this list probably doesn't contain all the optional values that some APIs would need. This is the spitballing phase!)_

While it might be neat to do that at some point, it would be pretty intimidating to the user. That's why I still think having at least a handful of built-in services that can be set by `pastebin_service` (for example) in the config is the best immediate option. Maybe later, if the user-defined pastebin idea happens, it could just be enabled by `pastebin_service = custom` or similar.
> > These applications are CLI utilities that let you submit stuff to various pastebin services. Integrating one with Sopel would relieve us from having to maintain a list of working pastebins and the methods to use them. But alas.
> 
> Alas, indeed. It would probably also relieve us of ever supporting Windows properly. Maybe even macOS.
> 

Fair point. Worries like that are the reason I've been looking if it's possible to get something like that as, say, a bundled python library rather than an external process (I agree that would be too much, especially since it adds a dependency), but no luck. I'll just stop making things complicated and add a handful of pastebin services for the user to choose from for now.
It really is the simplest option to just hardcode a handful of services for the redundancy. Some more complicated solution can come later if there's demand. :)

I look forward to reviewing your PR!

Resolving patch:
diff --git a/sopel/modules/help.py b/sopel/modules/help.py
--- a/sopel/modules/help.py
+++ b/sopel/modules/help.py
@@ -4,6 +4,8 @@
 Copyright 2008, Sean B. Palmer, inamidst.com
 Copyright © 2013, Elad Alfassa, <elad@fedoraproject.org>
 Copyright © 2018, Adam Erdman, pandorah.org
+Copyright © 2019, Tomasz Kurcz, github.com/uint
+Copyright © 2019, dgw, technobabbl.es
 Licensed under the Eiffel Forum License 2.
 
 https://sopel.chat
@@ -11,19 +13,151 @@
 from __future__ import unicode_literals, absolute_import, print_function, division
 
 import collections
-import requests
+import socket
 import textwrap
 
+import requests
+
 from sopel.logger import get_logger
 from sopel.module import commands, rule, example, priority
+from sopel.config.types import (
+    StaticSection, ChoiceAttribute
+)
 
 
 LOGGER = get_logger(__name__)
 
 
+class PostingException(Exception):
+    """Custom exception type for errors posting help to the chosen pastebin."""
+    pass
+
+
+# Pastebin handlers
+
+
+def _requests_post_catch_errors(*args, **kwargs):
+    try:
+        response = requests.post(*args, **kwargs)
+        response.raise_for_status()
+    except (
+            requests.exceptions.Timeout,
+            requests.exceptions.TooManyRedirects,
+            requests.exceptions.RequestException,
+            requests.exceptions.HTTPError
+    ):
+        # We re-raise all expected exception types to a generic "posting error"
+        # that's easy for callers to expect, and then we pass the original
+        # exception through to provide some debugging info
+        LOGGER.exception('Error during POST request')
+        raise PostingException('Could not communicate with remote service')
+
+    # remaining handling (e.g. errors inside the response) is left to the caller
+    return response
+
+
+def post_to_clbin(msg):
+    try:
+        result = _requests_post_catch_errors('https://clbin.com/', data={'clbin': msg})
+    except PostingException:
+        raise
+
+    result = result.text
+    if '://clbin.com/' in result:
+        # find/replace just in case the site tries to be sneaky and save on SSL overhead,
+        # though it will probably send us an HTTPS link without any tricks.
+        return result.replace('http://', 'https://', 1)
+    else:
+        LOGGER.error("Invalid result %s", result)
+        raise PostingException('clbin result did not contain expected URL base.')
+
+
+def post_to_0x0(msg):
+    try:
+        result = _requests_post_catch_errors('https://0x0.st', files={'file': msg})
+    except PostingException:
+        raise
+
+    result = result.text
+    if '://0x0.st' in result:
+        # find/replace just in case the site tries to be sneaky and save on SSL overhead,
+        # though it will probably send us an HTTPS link without any tricks.
+        return result.replace('http://', 'https://', 1)
+    else:
+        LOGGER.error('Invalid result %s', result)
+        raise PostingException('0x0.st result did not contain expected URL base.')
+
+
+def post_to_hastebin(msg):
+    try:
+        result = _requests_post_catch_errors('https://hastebin.com/documents', data=msg)
+    except PostingException:
+        raise
+
+    try:
+        result = result.json()
+    except ValueError:
+        LOGGER.error("Invalid Hastebin response %s", result)
+        raise PostingException('Could not parse response from Hastebin!')
+
+    if 'key' not in result:
+        LOGGER.error("Invalid result %s", result)
+        raise PostingException('Hastebin result did not contain expected URL base.')
+    return "https://hastebin.com/" + result['key']
+
+
+def post_to_termbin(msg):
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    sock.settimeout(10)  # the bot may NOT wait forever for a response; that would be bad
+    try:
+        sock.connect(('termbin.com', 9999))
+        sock.sendall(msg)
+        sock.shutdown(socket.SHUT_WR)
+        response = ""
+        while 1:
+            data = sock.recv(1024)
+            if data == "":
+                break
+            response += data
+        sock.close()
+    except socket.error:
+        LOGGER.exception('Error during communication with termbin')
+        raise PostingException('Error uploading to termbin')
+
+    # find/replace just in case the site tries to be sneaky and save on SSL overhead,
+    # though it will probably send us an HTTPS link without any tricks.
+    return response.strip('\x00\n').replace('http://', 'https://', 1)
+
+
+PASTEBIN_PROVIDERS = {
+    'clbin': post_to_clbin,
+    '0x0': post_to_0x0,
+    'hastebin': post_to_hastebin,
+    'termbin': post_to_termbin,
+}
+
+
+class HelpSection(StaticSection):
+    """Configuration section for this module."""
+    output = ChoiceAttribute('output',
+                             list(PASTEBIN_PROVIDERS),
+                             default='clbin')
+    """The pastebin provider to use for help output."""
+
+
+def configure(config):
+    config.define_section('help', HelpSection)
+    provider_list = ', '.join(PASTEBIN_PROVIDERS)
+    config.help.configure_setting(
+        'output',
+        'Pick a pastebin provider: {}: '.format(provider_list)
+    )
+
+
 def setup(bot):
     global help_prefix
     help_prefix = bot.config.core.help_prefix
+    bot.config.define_section('help', HelpSection)
 
 
 @rule('$nick' r'(?i)(help|doc) +([A-Za-z]+)(?:\?+)?$')
@@ -31,7 +165,7 @@ def setup(bot):
 @commands('help', 'commands')
 @priority('low')
 def help(bot, trigger):
-    """Shows a command's documentation, and possibly an example."""
+    """Shows a command's documentation, and an example if available. With no arguments, lists all commands."""
     if trigger.group(2):
         name = trigger.group(2)
         name = name.lower()
@@ -58,7 +192,11 @@ def msgfun(l):
         # actually creating the list first. Maybe worth storing the link and a
         # heuristic in the DB, too, so it persists across restarts. Would need a
         # command to regenerate, too...
-        if 'command-list' in bot.memory and bot.memory['command-list'][0] == len(bot.command_groups):
+        if (
+                'command-list' in bot.memory and
+                bot.memory['command-list'][0] == len(bot.command_groups) and
+                bot.memory['command-list'][2] == bot.config.help.output
+        ):
             url = bot.memory['command-list'][1]
         else:
             bot.say("Hang on, I'm creating a list.")
@@ -77,28 +215,28 @@ def msgfun(l):
             url = create_list(bot, '\n\n'.join(msgs))
             if not url:
                 return
-            bot.memory['command-list'] = (len(bot.command_groups), url)
+            bot.memory['command-list'] = (len(bot.command_groups),
+                                          url,
+                                          bot.config.help.output)
         bot.say("I've posted a list of my commands at {0} - You can see "
                 "more info about any of these commands by doing {1}help "
                 "<command> (e.g. {1}help time)".format(url, help_prefix))
 
 
 def create_list(bot, msg):
+    """Creates & uploads the command list.
+
+    Returns the URL from the chosen pastebin provider.
+    """
     msg = 'Command listing for {}@{}\n\n'.format(bot.nick, bot.config.core.host) + msg
 
     try:
-        result = requests.post('https://clbin.com/', data={'clbin': msg})
-    except requests.RequestException:
+        result = PASTEBIN_PROVIDERS[bot.config.help.output](msg)
+    except PostingException:
         bot.say("Sorry! Something went wrong.")
         LOGGER.exception("Error posting commands")
         return
-    result = result.text
-    if "https://clbin.com/" in result:
-        return result
-    else:
-        bot.say("Sorry! Something went wrong.")
-        LOGGER.error("Invalid result %s", result)
-        return
+    return result
 
 
 @rule('$nick' r'(?i)help(?:[?!]+)?$')


Test patch: