Repository: sopel-irc/sopel

Issue:
[core]: Unicode issue in long messages
There is a bug concerning unicode characters when a message is long enough.

Here, where the message is short, everything works correctly:

<pre>
Erethon: !in 10s τεστ
BuildTest: Erethon: Okay, will remind in 10 secs
BuildTest: Erethon: τεστ
</pre>


But, when the message is long enough:

<pre>
Erethon: !in 10s 
αααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααα
BuildTest: Erethon: Okay, will remind in 10 secs
BuildTest: Erethon: 
Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î
</pre>


This is marked as [core] because it happens in more than one module, I've confirmed it in ".py" too.
I'm using python 2.7.3 and latest Willie version from master branch and I've installed Willie by running 'python setup.py install'.



Discussion:
I'll take a look later

I can't reproduce this issue. Tried with `.in` and with `.py`, with both Greek and Hebrew letters.

I do get this warning however
`
/home/elad/Sources/willie/willie/irc.py:474: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
`

Will investigate further.

Find a reproducing case:

```
<elad> .py (u"α"*10000).encode('utf-8')
<eladtestbot> Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±Î±
```

reproduces with short messages too, no idea why, seems specific to .py tho.

Having a message quite similar with rss.py (i'm using the last version ﻿[8cd3bef](https://github.com/embolalia/willie/commit/8cd3bef40c5e3a507c76352f6c09ad542734a696))
`/usr/local/lib/python2.7/dist-packages/willie/modules/rss.py:449: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if (feed.title == entry.title and feed.link == entry.link`

This has nothing to do with this issue, please open a separate one

Sorry @elad661 . Here we [go](https://github.com/embolalia/willie/issues/543).

I've had this happen when the message sent to the IRC server is trimmed in between bytes of the unicode characters. Maybe caused by https://github.com/sopel-irc/sopel/blob/master/sopel/bot.py#L280-L300

Indeed, @maxpowa was right!

See this example:

```
>>> len('α')
1
>>> len('α'.encode('utf-8'))
2
```

To fix this, I'll provide a PR where I loop over the unicode string until the bytecode version is smaller.

Resolving patch:
diff --git a/sopel/bot.py b/sopel/bot.py
--- a/sopel/bot.py
+++ b/sopel/bot.py
@@ -290,27 +290,15 @@ def say(self, text, recipient, max_messages=1):
         message will contain the entire remainder, which may be truncated by
         the server.
         """
-        # We're arbitrarily saying that the max is 400 bytes of text when
-        # messages will be split. Otherwise, we'd have to acocunt for the bot's
-        # hostmask, which is hard.
-        max_text_length = 400
-        # Encode to bytes, for propper length calculation
-        if isinstance(text, unicode):
-            encoded_text = text.encode('utf-8')
-        else:
-            encoded_text = text
         excess = ''
-        if max_messages > 1 and len(encoded_text) > max_text_length:
-            last_space = encoded_text.rfind(' '.encode('utf-8'), 0, max_text_length)
-            if last_space == -1:
-                excess = encoded_text[max_text_length:]
-                encoded_text = encoded_text[:max_text_length]
-            else:
-                excess = encoded_text[last_space + 1:]
-                encoded_text = encoded_text[:last_space]
-        # We'll then send the excess at the end
-        # Back to unicode again, so we don't screw things up later.
-        text = encoded_text.decode('utf-8')
+        if not isinstance(text, unicode):
+            # Make sure we are dealing with unicode string
+            text = text.decode('utf-8')
+
+        if max_messages > 1:
+            # Manage multi-line only when needed
+            text, excess = tools.get_sendable_message(text)
+
         try:
             self.sending.acquire()
 
diff --git a/sopel/irc.py b/sopel/irc.py
--- a/sopel/irc.py
+++ b/sopel/irc.py
@@ -138,13 +138,24 @@ def write(self, args, text=None):
             # CR-LF (Carriage Return - Line Feed) pair, and these messages SHALL
             # NOT exceed 512 characters in length, counting all characters
             # including the trailing CR-LF. Thus, there are 510 characters
-            # maximum allowed for the command and its parameters.  There is no
+            # maximum allowed for the command and its parameters. There is no
             # provision for continuation of message lines.
 
+            max_length = unicode_max_length = 510
             if text is not None:
-                temp = (' '.join(args) + ' :' + text)[:510] + '\r\n'
+                temp = (' '.join(args) + ' :' + text)
             else:
-                temp = ' '.join(args)[:510] + '\r\n'
+                temp = ' '.join(args)
+
+            # The max length of 512 is in bytes, not unicode
+            while len(temp.encode('utf-8')) > max_length:
+                temp = temp[:unicode_max_length]
+                unicode_max_length = unicode_max_length - 1
+
+            # Ends the message with CR-LF
+            temp = temp + '\r\n'
+
+            # Log and output the message
             self.log_raw(temp, '>>')
             self.send(temp.encode('utf-8'))
         finally:
diff --git a/sopel/tools/__init__.py b/sopel/tools/__init__.py
--- a/sopel/tools/__init__.py
+++ b/sopel/tools/__init__.py
@@ -152,6 +152,40 @@ def get_nickname_command_pattern(command):
         """.format(command=command)
 
 
+def get_sendable_message(text, max_length=400):
+    """Get a sendable ``text`` message, with its excess when needed.
+
+    :param str txt: unicode string of text to send
+    :param int max_length: maximum length of the message to be sendable
+    :return: a tuple of two values, the sendable text and its excess text
+
+    We're arbitrarily saying that the max is 400 bytes of text when
+    messages will be split. Otherwise, we'd have to account for the bot's
+    hostmask, which is hard.
+
+    The `max_length` is the max length of text in **bytes**, but we take
+    care of unicode 2-bytes characters, by working on the unicode string,
+    then making sure the bytes version is smaller than the max length.
+    """
+    unicode_max_length = max_length
+    excess = ''
+
+    while len(text.encode('utf-8')) > max_length:
+        last_space = text.rfind(' ', 0, unicode_max_length)
+        if last_space == -1:
+            # No last space, just split where it is possible
+            excess = text[unicode_max_length:] + excess
+            text = text[:unicode_max_length]
+            # Decrease max length for the unicode string
+            unicode_max_length = unicode_max_length - 1
+        else:
+            # Split at the last best space found
+            excess = text[last_space:]
+            text = text[:last_space]
+
+    return text, excess.lstrip()
+
+
 def deprecated(old):
     def new(*args, **kwargs):
         print('Function %s is deprecated.' % old.__name__, file=sys.stderr)


Test patch:
diff --git a/test/test_tools.py b/test/test_tools.py
new file mode 100644
--- /dev/null
+++ b/test/test_tools.py
@@ -0,0 +1,93 @@
+# coding=utf-8
+"""Tests sopel.tools"""
+from __future__ import unicode_literals, absolute_import, print_function, division
+
+
+from sopel import tools
+
+
+def test_get_sendable_message_default():
+    initial = 'aaaa'
+    text, excess = tools.get_sendable_message(initial)
+
+    assert text == initial
+    assert excess == ''
+
+
+def test_get_sendable_message_limit():
+    initial = 'a' * 400
+    text, excess = tools.get_sendable_message(initial)
+
+    assert text == initial
+    assert excess == ''
+
+
+def test_get_sendable_message_excess():
+    initial = 'a' * 401
+    text, excess = tools.get_sendable_message(initial)
+
+    assert text == 'a' * 400
+    assert excess == 'a'
+
+
+def test_get_sendable_message_excess_space():
+    # aaa...aaa bbb...bbb
+    initial = ' '.join(['a' * 200, 'b' * 200])
+    text, excess = tools.get_sendable_message(initial)
+
+    assert text == 'a' * 200
+    assert excess == 'b' * 200
+
+
+def test_get_sendable_message_excess_space_limit():
+    # aaa...aaa bbb...bbb
+    initial = ' '.join(['a' * 400, 'b' * 200])
+    text, excess = tools.get_sendable_message(initial)
+
+    assert text == 'a' * 400
+    assert excess == 'b' * 200
+
+
+def test_get_sendable_message_excess_bigger():
+    # aaa...aaa bbb...bbb
+    initial = ' '.join(['a' * 401, 'b' * 1000])
+    text, excess = tools.get_sendable_message(initial)
+
+    assert text == 'a' * 400
+    assert excess == 'a ' + 'b' * 1000
+
+
+def test_get_sendable_message_optional():
+    text, excess = tools.get_sendable_message('aaaa', 3)
+    assert text == 'aaa'
+    assert excess == 'a'
+
+    text, excess = tools.get_sendable_message('aaa bbb', 3)
+    assert text == 'aaa'
+    assert excess == 'bbb'
+
+    text, excess = tools.get_sendable_message('aa bb cc', 3)
+    assert text == 'aa'
+    assert excess == 'bb cc'
+
+
+def test_get_sendable_message_two_bytes():
+    text, excess = tools.get_sendable_message('αααα', 4)
+    assert text == 'αα'
+    assert excess == 'αα'
+
+    text, excess = tools.get_sendable_message('αααα', 5)
+    assert text == 'αα'
+    assert excess == 'αα'
+
+    text, excess = tools.get_sendable_message('α ααα', 4)
+    assert text == 'α'
+    assert excess == 'ααα'
+
+    text, excess = tools.get_sendable_message('αα αα', 4)
+    assert text == 'αα'
+    assert excess == 'αα'
+
+    text, excess = tools.get_sendable_message('ααα α', 4)
+    assert text == 'αα'
+    assert excess == 'α α'