Repository: sopel-irc/sopel

Issue:
Identifier lowercase is not right?
Poking around in `Identifier`'s class definition for other reasons, I saw that `Identifier._lower()` makes the following replacements after lowercasing the input:

| Character | Replacement |
|:---------:|:-----------:|
|    `{`    |     `[`     |
|    `}`    |     `]`     |
|    `\|`   |     `\\`    |
|    `^`    |     `~`     |

RFC 2812 [defines](https://tools.ietf.org/html/rfc2812#section-2.2) the characters in the left column as "lowercase" and the characters in the right column as "uppercase", though, so it seems `Identifier._lower()` has been doing case conversion of these symbols in the wrong direction for years. :joy_cat:

Now, I don't think that changing this will actually affect anything at runtime. `bot.users` (and friends) gets populated at connection time, and will just use whatever lowercase representation that's generated.

The most likely wrinkle is from the bot database, which stores both the `canonical` nickname (capitalized as seen on IRC) and a `slug` representation (in lowercase). The generated `slug` might change for some nicks in the database after updating to a version of Sopel that changes `Identifier._lower()`, and so we might need a database migration script to fix that up. It's likely that changing the lowercase representation of `Identifier`s would result in some users with an affected character in their nick losing their settings unless a migration is run. (A unique `slug`—the primary key for the `nicknames` table—would generate a new `nick_id`, which wouldn't have any matching rows in `nick_values`.)

Changing how `Identifier`s work could also potentially break a few third-party modules—especially if they store `Identifier` values as lowercase in a custom database table or external file. For that reason, even though I would really rather just fix this for **6.6.0**, I think the changelog will simply include a notice that this change is planned for Sopel 7 so end users and module developers have time to plan ahead. (See task list below for why the milestone is not currently set to **7.0.0**.)

----

Task List:

  - [x] Note in all future Sopel 6.x release notes that this change is planned for Sopel 7.
    - Update milestone as appropriate to make sure this doesn't get forgotten.
  - [x] Actually fix this bug in Sopel 7. (#1744)


Discussion:
Pushing to next milestone. I've made a draft changelog for 6.6.0 and this reminder is no longer needed.

Resolving patch:
diff --git a/sopel/db.py b/sopel/db.py
--- a/sopel/db.py
+++ b/sopel/db.py
@@ -206,6 +206,16 @@ def get_nick_id(self, nick, create=True):
                 .one_or_none()
 
             if nickname is None:
+                # see if it needs case-mapping migration
+                nickname = session.query(Nicknames) \
+                    .filter(Nicknames.slug == Identifier._lower_swapped(nick)) \
+                    .one_or_none()
+                if nickname is not None:
+                    # it does!
+                    nickname.slug = slug
+                    session.commit()
+
+            if nickname is None:  # "is /* still */ None", if Python had inline comments
                 if not create:
                     raise ValueError('No ID exists for the given nick')
                 # Generate a new ID
@@ -390,9 +400,47 @@ def merge_nick_groups(self, first_nick, second_nick):
 
     # CHANNEL FUNCTIONS
 
+    def get_channel_slug(self, chan):
+        """Return the case-normalized representation of ``channel``.
+
+        :param str channel: the channel name to normalize, with prefix
+                            (required)
+        :return str: the case-normalized channel name (or "slug"
+                     representation)
+
+        This is useful to make sure that a channel name is stored consistently
+        in both the bot's own database and third-party plugins'
+        databases/files, without regard for variation in case between
+        different clients and/or servers on the network.
+        """
+        chan = Identifier(chan)
+        slug = chan.lower()
+        session = self.ssession()
+        try:
+            count = session.query(ChannelValues) \
+                .filter(ChannelValues.channel == slug) \
+                .count()
+
+            if count == 0:
+                # see if it needs case-mapping migration
+                old_rows = session.query(ChannelValues) \
+                    .filter(ChannelValues.channel == Identifier._lower_swapped(chan))
+                old_count = old_rows.count()
+                if old_count > 0:
+                    # it does!
+                    old_rows.update({ChannelValues.channel: slug})
+                    session.commit()
+
+            return slug
+        except SQLAlchemyError:
+            session.rollback()
+            raise
+        finally:
+            self.ssession.remove()
+
     def set_channel_value(self, channel, key, value):
         """Sets the value for a given key to be associated with the channel."""
-        channel = Identifier(channel).lower()
+        channel = self.get_channel_slug(channel)
         value = json.dumps(value, ensure_ascii=False)
         session = self.ssession()
         try:
@@ -417,7 +465,7 @@ def set_channel_value(self, channel, key, value):
 
     def delete_channel_value(self, channel, key):
         """Deletes the value for a given key associated with a channel."""
-        channel = Identifier(channel).lower()
+        channel = self.get_channel_slug(channel)
         session = self.ssession()
         try:
             result = session.query(ChannelValues) \
@@ -436,7 +484,7 @@ def delete_channel_value(self, channel, key):
 
     def get_channel_value(self, channel, key):
         """Retrieves the value for a given key associated with a channel."""
-        channel = Identifier(channel).lower()
+        channel = self.get_channel_slug(channel)
         session = self.ssession()
         try:
             result = session.query(ChannelValues) \
diff --git a/sopel/tools/__init__.py b/sopel/tools/__init__.py
--- a/sopel/tools/__init__.py
+++ b/sopel/tools/__init__.py
@@ -420,6 +420,26 @@ def _lower(identifier):
             return identifier._lowered
         # The tilde replacement isn't needed for identifiers, but is for
         # channels, which may be useful at some point in the future.
+        low = identifier.lower().replace('[', '{').replace(']', '}')
+        low = low.replace('\\', '|').replace('~', '^')
+        return low
+
+    @staticmethod
+    def _lower_swapped(identifier):
+        """Backward-compatible version of :meth:`_lower`.
+
+        :param str identifier: the identifier (nickname or channel) to convert
+        :return: RFC 2812-non-compliant lowercase version of ``identifier``
+        :rtype str:
+
+        This is what the old ``_lower()`` function did before Sopel 7.0. It maps
+        ``{}``, ``[]``, ``|``, ``\\``, ``^``, and ``~`` incorrectly.
+
+        You shouldn't use this unless you need to migrate stored values from the
+        previous, incorrect "lowercase" representation to the correct one.
+        """
+        # The tilde replacement isn't needed for identifiers, but is for
+        # channels, which may be useful at some point in the future.
         low = identifier.lower().replace('{', '[').replace('}', ']')
         low = low.replace('|', '\\').replace('^', '~')
         return low


Test patch:
diff --git a/test/test_db.py b/test/test_db.py
--- a/test/test_db.py
+++ b/test/test_db.py
@@ -49,7 +49,7 @@ def test_get_nick_id(db):
     tests = [
         [None, 'embolalia', Identifier('Embolalia')],
         # Ensures case conversion is handled properly
-        [None, '[][]', Identifier('[]{}')],
+        [None, '{}{}', Identifier('[]{}')],
         # Unicode, just in case
         [None, 'embölaliå', Identifier('EmbölaliÅ')],
     ]