Repository: sopel-irc/sopel

Issue:
A general way to truncate messages and stay within the maximum length
After being rightfully scolded by @dgw for hijacking another issue, I am opening a separate one.

It would be nice to have a tool that automatically cuts off the bot responses which are going to be too long, adds a "[…]" at the end, and continues in private message or adds a web link.

There are a few modules which use their own logic to accomplish this.
Sopel should keep track of its full hostmask
Can we properly setup `Sopel.say()` to construct PRIVMSGs that will actually be at the 512 limit?


Discussion:
The bot needs to know the length of its hostmask to do this, so this should be implemented with #1322.
This is actually harder to do than it sounds. I'd like for Sopel to use the full line length available, too, but external factors can change an IRC user's hostname without notice: server-side mode changes, HostServ or similar being turned on/off, cloaking applied after NickServ authentication, etc.

Is there a reliable way to monitor for changes in one's own hostname that I'm unaware of? Happy to look into doing this if it's technically possible without a kludge like polling. :smile_cat:

As an aside, I've used multiple other IRC clients and frameworks that conservatively truncate (split) lines because they don't want to deal with tracking hostname changes either.
Sopel tracks its hostmask as of #1537, but this is left open for the "512 limit" piece—that is, making Sopel actually use the stored hostmask to send as much as possible.
Just going to leave this here in regards to how many bytes the hostmask takes, and how many bytes it could portentially take as error handling.


````
        try:
            hostmaskbytes = len((bot.users.get(bot.nick).hostmask).encode('utf-8'))
        except AttributeError:
            hostmaskbytes = (len((bot.nick).encode('utf-8'))  # Bot's NICKLEN
                             + 1  # (! separator)
                             + len('~')  # (for the optional ~ in user)
                             + 9  # max username length
                             + 1  # (@ separator)
                             + 63  # <hostname> has a maximum length of 63 characters.
                             )
````
@deathbybandaid In 7.x we have `bot.hostmask`, FYI.

Resolving patch:
diff --git a/sopel/bot.py b/sopel/bot.py
--- a/sopel/bot.py
+++ b/sopel/bot.py
@@ -1118,7 +1118,7 @@ def __getattr__(self, attr):
     def __setattr__(self, attr, value):
         return setattr(self._bot, attr, value)
 
-    def say(self, message, destination=None, max_messages=1):
+    def say(self, message, destination=None, max_messages=1, trailing=''):
         """Override ``Sopel.say`` to use trigger source by default.
 
         :param str message: message to say
@@ -1136,7 +1136,7 @@ def say(self, message, destination=None, max_messages=1):
         """
         if destination is None:
             destination = self._trigger.sender
-        self._bot.say(self._out_pfx + message, destination, max_messages)
+        self._bot.say(self._out_pfx + message, destination, max_messages, trailing)
 
     def action(self, message, destination=None):
         """Override ``Sopel.action`` to use trigger source by default.
diff --git a/sopel/irc/__init__.py b/sopel/irc/__init__.py
--- a/sopel/irc/__init__.py
+++ b/sopel/irc/__init__.py
@@ -533,34 +533,76 @@ def reply(self, text, dest, reply_to, notice=False):
         else:
             self.say(text, dest)
 
-    def say(self, text, recipient, max_messages=1):
+    def say(self, text, recipient, max_messages=1, trailing=''):
         """Send a PRIVMSG to a user or channel.
 
         :param str text: the text to send
         :param str recipient: the message recipient
         :param int max_messages: split ``text`` into at most this many messages
                                  if it is too long to fit in one (optional)
+        :param str trailing: text to append if ``text`` is too long to fit in
+                             a single message, or into the last message if
+                             ``max_messages`` is greater than 1 (optional)
 
         By default, this will attempt to send the entire ``text`` in one
         message. If the text is too long for the server, it may be truncated.
 
         If ``max_messages`` is given, the ``text`` will be split into at most
-        that many messages, each no more than 400 bytes. The split is made at
-        the last space character before the 400th byte, or at the 400th byte
-        if no such space exists.
+        that many messages. The split is made at the last space character
+        before the "safe length" (which is calculated based on the bot's
+        nickname and hostmask), or exactly at the "safe length" if no such
+        space character exists.
 
         If the ``text`` is too long to fit into the specified number of
         messages using the above splitting, the final message will contain the
         entire remainder, which may be truncated by the server.
+
+        The ``trailing`` parameter allows gracefully terminating a ``text``
+        that is too long to fit in the specified number of messages. The final
+        message (or only message, if ``max_messages`` is left at the default
+        value of 1) will be truncated slightly to fit the ``trailing`` string.
+        Note that the ``trailing`` parameter must include leading whitespace
+        if you desire any between it and the truncated text.
         """
         excess = ''
         if not isinstance(text, unicode):
             # Make sure we are dealing with a Unicode string
             text = text.decode('utf-8')
 
-        if max_messages > 1:
-            # Manage multi-line only when needed
-            text, excess = tools.get_sendable_message(text)
+        if max_messages > 1 or trailing:
+            # Handle message splitting/truncation only if needed
+            try:
+                hostmask_length = len(self.hostmask)
+            except KeyError:
+                # calculate maximum possible length, given current nick/username
+                hostmask_length = (
+                    len(self.nick)  # own nick length
+                    + 1  # (! separator)
+                    + 1  # (for the optional ~ in user)
+                    + min(  # own ident length, capped to ISUPPORT or RFC maximum
+                        len(self.user),
+                        getattr(self.isupport, 'USERLEN', 9))
+                    + 1  # (@ separator)
+                    + 63  # <hostname> has a maximum length of 63 characters.
+                )
+            safe_length = (
+                512  # maximum IRC line length in bytes, per RFC
+                - 1  # leading colon
+                - hostmask_length  # calculated/maximum length of own hostmask prefix
+                - 1  # space between prefix & command
+                - 7  # PRIVMSG command
+                - 1  # space before recipient
+                - len(recipient.encode('utf-8'))  # target channel/nick (can contain Unicode)
+                - 2  # space after recipient, colon before text
+                - 2  # trailing CRLF
+            )
+            text, excess = tools.get_sendable_message(text, safe_length)
+
+        if max_messages == 1 and excess and trailing:
+            # only append `trailing` if this is the last message AND it's still too long
+            safe_length -= len(trailing.encode('utf-8'))
+            text, excess = tools.get_sendable_message(text, safe_length)
+            text += trailing
 
         flood_max_wait = self.settings.core.flood_max_wait
         flood_burst_lines = self.settings.core.flood_burst_lines
@@ -635,7 +677,7 @@ def say(self, text, recipient, max_messages=1):
             recipient_stack['messages'].append((time.time(), safe(text)))
             recipient_stack['messages'] = recipient_stack['messages'][-10:]
 
-        # Now that we've sent the first part, we need to send the rest. Doing
-        # this recursively seems easier to me than iteratively
-        if excess:
-            self.say(excess, recipient, max_messages - 1)
+        # Now that we've sent the first part, we need to send the rest if
+        # requested. Doing so recursively seems simpler than iteratively.
+        if max_messages > 1 and excess:
+            self.say(excess, recipient, max_messages - 1, trailing)


Test patch:
diff --git a/test/test_irc.py b/test/test_irc.py
--- a/test/test_irc.py
+++ b/test/test_irc.py
@@ -28,6 +28,11 @@ def bot(tmpconfig, botfactory):
     return botfactory(tmpconfig)
 
 
+def prefix_length(bot):
+    # ':', nick, '!', '~', ident/username, '@', maximum hostname length, <0x20>
+    return 1 + len(bot.nick) + 1 + 1 + len(bot.user) + 1 + 63 + 1
+
+
 def test_on_connect(bot):
     bot.on_connect()
 
@@ -241,7 +246,7 @@ def test_say_long_extra(bot):
 
 def test_say_long_extra_multi_message(bot):
     """Test a long message that doesn't fit, with split allowed."""
-    text = 'a' * 400
+    text = 'a' * (512 - prefix_length(bot) - len('PRIVMSG #sopel :\r\n'))
     bot.say(text + 'b', '#sopel', max_messages=2)
 
     assert bot.backend.message_sent == rawlist(
@@ -250,6 +255,67 @@ def test_say_long_extra_multi_message(bot):
     )
 
 
+def test_say_long_extra_multi_message_multibyte_recipient(bot):
+    """Test a split-allowed message sent to recipient with multi-byte char."""
+    text = 'a' * (
+        512 - prefix_length(bot) - len('PRIVMSG #sopel² :\r\n'.encode('utf-8')))
+    bot.say(text + 'b', '#sopel²', max_messages=2)
+
+    assert bot.backend.message_sent == rawlist(
+        'PRIVMSG #sopel² :%s' % text,  # the 'b' is split from message
+        'PRIVMSG #sopel² :b',
+    )
+
+
+def test_say_long_trailing_fit(bot):
+    """Test optional truncation indicator with message that fits in one line."""
+    text = 'a' * (512 - prefix_length(bot) - len('PRIVMSG #sopel :\r\n') - 3)
+    bot.say(text + 'ttt', '#sopel', trailing='...')
+
+    assert bot.backend.message_sent == rawlist(
+        'PRIVMSG #sopel :%s' % text + 'ttt',  # nothing is truncated or replaced
+    )
+
+
+def test_say_long_trailing_extra(bot):
+    """Test optional truncation indicator with message that is too long."""
+    text = 'a' * (512 - prefix_length(bot) - len('PRIVMSG #sopel :\r\n') - 3)
+    bot.say(text + 'ttt' + 'b', '#sopel', trailing='...')
+
+    assert bot.backend.message_sent == rawlist(
+        # 'b' is truncated; 'ttt' is replaced by `trailing`
+        'PRIVMSG #sopel :%s' % text + '...',
+    )
+
+
+def test_say_long_trailing_extra_multi_message(bot):
+    """Test optional truncation indicator with message splitting."""
+    msg1 = 'a' * (512 - prefix_length(bot) - len('PRIVMSG #sopel :\r\n'))
+    msg2 = msg1[:-3] + 'ttt'
+    bot.say(msg1 + msg2 + 'b', '#sopel', max_messages=2, trailing='...')
+
+    assert bot.backend.message_sent == rawlist(
+        # split as expected
+        'PRIVMSG #sopel :%s' % msg1,
+        # 'b' is truncated; 'ttt' is replaced by `trailing`
+        'PRIVMSG #sopel :%s' % msg2.replace('ttt', '...'),
+    )
+
+
+def test_say_long_trailing_extra_multi_message_multibyte(bot):
+    """Test optional truncation indicator with message splitting."""
+    msg1 = 'a' * (512 - prefix_length(bot) - len('PRIVMSG #sopel :\r\n'))
+    msg2 = msg1[:-3] + 'ttt'
+    bot.say(msg1 + msg2 + 'b', '#sopel', max_messages=2, trailing='…')
+
+    assert bot.backend.message_sent == rawlist(
+        # split as expected
+        'PRIVMSG #sopel :%s' % msg1,
+        # 'b' is truncated; 'ttt' is replaced by `trailing`
+        'PRIVMSG #sopel :%s' % msg2.replace('ttt', '…'),
+    )
+
+
 def test_say_no_repeat_protection(bot):
     # five is fine
     bot.say('hello', '#sopel')