GitHub PullRequestReviewCommentEvent at 2014-03-28 09:29:52
Repository: picard
Actor: Sophist-UK
Payload: {"comment":{"url":"https://api.github.com/repos/musicbrainz/picard/pulls/comments/11059203","id":11059203,"diff_hunk":"@@ -0,0 +1,439 @@\n+# -*- coding: utf-8 -*-\n+#\n+# Picard, the next-generation MusicBrainz tagger\n+# Copyright (C) 2004 Robert Kaye\n+# Copyright (C) 2006 Lukáš Lalinský\n+#\n+# This program is free software; you can redistribute it and/or\n+# modify it under the terms of the GNU General Public License\n+# as published by the Free Software Foundation; either version 2\n+# of the License, or (at your option) any later version.\n+#\n+# This program is distributed in the hope that it will be useful,\n+# but WITHOUT ANY WARRANTY; without even the implied warranty of\n+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n+# GNU General Public License for more details.\n+#\n+# You should have received a copy of the GNU General Public License\n+# along with this program; if not, write to the Free Software\n+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.\n+\n+# This modules provides functionality for simplifying unicode strings.\n+\n+# The unicode character set (of over 1m codepoints and 24,000 characters) includes:\n+#   Normal ascii (latin) non-accented characters\n+#   Combined latin characters e.g. ae in normal usage\n+#   Compatibility combined latin characters (retained for compatibility with other character sets)\n+#     These can look very similar to normal characters and can be confusing for searches, sort orders etc.\n+#   Non-latin (e.g. japanese, greek, hebrew etc.) characters\n+# Both latin and non-latin characters can be accented. Accents can be either:\n+#   Provided by separate nonspacing_mark characters which are visually overlaid (visually 1 character is actually 2); or\n+#   Integrated accented characters (i.e. non-accented characters combined with a nonspace_mark into a single character)\n+# Again these can be confusing for searches, sort orders etc.\n+# Punctuation can also be confusing in unicode e.g. several types of single or double quote mark.\n+\n+# For latin script:\n+#   Combined characters, accents and punctuation can be visually similar but look different to search engines,\n+#   sort orders etc. and the number of ways to use similar looking characters can (does) result in inconsistent\n+#   usage inside Music metadata.\n+#\n+#   Simplifying # the unicode character sets by many-to-one mappings can improve consistency and reduce confusion,\n+#   however sometimes the choice of specific characters can be a deliberate part of an album, song title or artist name\n+#   (and should not therefore be changed without careful thought) and occasionally the choice of characters can be\n+#   malicious (i.e. to defeat firewalls or spam filters or to appear to be something else).\n+#\n+#   Finally, given the size of the unicode character set, fonts are unlikely to display all characters,\n+#   making simplification a necessity.\n+#\n+#   Simplification may also be needed to make tags conform to ISO-8859-1 (extended ascii) or to make tags or filenames\n+#   into ascii, perhaps because the file system or player cannot support unicode.\n+#\n+# Non-latin scripts may also need to be converted to latin scripts through:\n+#   Translation (e.g. hebrew word for mother is translated to \"mother\"); or\n+#   Transliteration (e.g. the SOUND of the hebrew letter or word is spelt out in latin)\n+# These are non-trivial, and the software to do these is far from comprehensive.\n+\n+# This module provides utility functions to enable simplification of latin and punctuation unicode:\n+#   1. simplify compatibility characters;\n+#   2. split combined characters;\n+#   3. remove accents (entirely or if not in ISO-8859-1 as applicable);\n+#   4. replace remaining non-ascii or non-ISO-8859-1 characters with a default character\n+# This module also provides an extension infrastructure to allow translation and / or transliteration plugins to be added.\n+\n+import re\n+import unicodedata\n+import codecs\n+from functools import partial\n+\n+#########################  LATIN SIMPLIFICATION ###########################\n+# The translation tables for punctuation and latin combined-characters are taken from\n+# http://unicode.org/repos/cldr/trunk/common/transforms/Latin-ASCII.xml\n+# Various bugs and mistakes in this have been ironed out during testing.\n+\n+\n+def _re_any(iterable):\n+    return re.compile('([' + ''.join(iterable) + '])', re.UNICODE)\n+\n+_additional_compatibility = {\n+    u\"\\u0276\": u\"Œ\",  # LATIN LETTER SMALL CAPITAL OE\n+    u\"\\u1D00\": u\"A\",  # LATIN LETTER SMALL CAPITAL A\n+    u\"\\u1D01\": u\"Æ\",  # LATIN LETTER SMALL CAPITAL AE\n+    u\"\\u1D04\": u\"C\",  # LATIN LETTER SMALL CAPITAL C\n+    u\"\\u1D05\": u\"D\",  # LATIN LETTER SMALL CAPITAL D\n+    u\"\\u1D07\": u\"E\",  # LATIN LETTER SMALL CAPITAL E\n+    u\"\\u1D0A\": u\"J\",  # LATIN LETTER SMALL CAPITAL J\n+    u\"\\u1D0B\": u\"K\",  # LATIN LETTER SMALL CAPITAL K\n+    u\"\\u1D0D\": u\"M\",  # LATIN LETTER SMALL CAPITAL M\n+    u\"\\u1D0F\": u\"O\",  # LATIN LETTER SMALL CAPITAL O\n+    u\"\\u1D18\": u\"P\",  # LATIN LETTER SMALL CAPITAL P\n+    u\"\\u1D1B\": u\"T\",  # LATIN LETTER SMALL CAPITAL T\n+    u\"\\u1D1C\": u\"U\",  # LATIN LETTER SMALL CAPITAL U\n+    u\"\\u1D20\": u\"V\",  # LATIN LETTER SMALL CAPITAL V\n+    u\"\\u1D21\": u\"W\",  # LATIN LETTER SMALL CAPITAL W\n+    u\"\\u1D22\": u\"Z\",  # LATIN LETTER SMALL CAPITAL Z\n+    u\"\\u3007\": u\"0\",  # IDEOGRAPHIC NUMBER ZERO\n+    u\"\\u00A0\": u\" \",  # NO-BREAK SPACE\n+    u\"\\u3000\": u\" \",  # IDEOGRAPHIC SPACE (from ‹character-fallback›)\n+    u\"\\u2033\": u\"”\",  # DOUBLE PRIME\n+}\n+_re_additional_compatibility = _re_any(_additional_compatibility.keys())\n+\n+def unicode_simplify_compatibility(string):\n+    interim = _re_additional_compatibility.sub(lambda m: _additional_compatibility[m.group(0)], string)\n+    return unicodedata.normalize(\"NFKC\", interim)\n+\n+\n+_simplify_punctuation = {\n+    u\"\\u013F\": u\"L\",  # LATIN CAPITAL LETTER L WITH MIDDLE DOT (compat)\n+    u\"\\u0140\": u\"l\",  # LATIN SMALL LETTER L WITH MIDDLE DOT (compat)\n+    u\"\\u2018\": u\"'\",  # LEFT SINGLE QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u2019\": u\"'\",  # RIGHT SINGLE QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u201A\": u\",\",  # SINGLE LOW-9 QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u201B\": u\"'\",  # SINGLE HIGH-REVERSED-9 QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u201C\": u\"\\\"\",  # LEFT DOUBLE QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u201D\": u\"\\\"\",  # RIGHT DOUBLE QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u201E\": u\",,\",  # DOUBLE LOW-9 QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u201F\": u\"\\\"\",  # DOUBLE HIGH-REVERSED-9 QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u2032\": u\"'\",  # PRIME\n+    u\"\\u2033\": u\"\\\"\",  # DOUBLE PRIME\n+    u\"\\u301D\": u\"\\\"\",  # REVERSED DOUBLE PRIME QUOTATION MARK\n+    u\"\\u301E\": u\"\\\"\",  # DOUBLE PRIME QUOTATION MARK\n+    u\"\\u00AB\": u\"<<\",  # LEFT-POINTING DOUBLE ANGLE QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u00BB\": u\">>\",  # RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK (from ‹character-fallback›)\n+    u\"\\u2039\": u\"<\",  # SINGLE LEFT-POINTING ANGLE QUOTATION MARK\n+    u\"\\u203A\": u\">\",  # SINGLE RIGHT-POINTING ANGLE QUOTATION MARK\n+    u\"\\u00AD\": u\"-\",  # SOFT HYPHEN (from ‹character-fallback›)","path":"picard/util/textencoding.py","position":125,"original_position":125,"commit_id":"7ea0ec4db98a27d44132051a44ec473ed3734020","original_commit_id":"7ea0ec4db98a27d44132051a44ec473ed3734020","user":{"login":"Sophist-UK","id":3001893,"avatar_url":"https://avatars.githubusercontent.com/u/3001893?","gravatar_id":"5ee62c0cbbbe6887a768a34255453203","url":"https://api.github.com/users/Sophist-UK","html_url":"https://github.com/Sophist-UK","followers_url":"https://api.github.com/users/Sophist-UK/followers","following_url":"https://api.github.com/users/Sophist-UK/following{/other_user}","gists_url":"https://api.github.com/users/Sophist-UK/gists{/gist_id}","starred_url":"https://api.github.com/users/Sophist-UK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Sophist-UK/subscriptions","organizations_url":"https://api.github.com/users/Sophist-UK/orgs","repos_url":"https://api.github.com/users/Sophist-UK/repos","events_url":"https://api.github.com/users/Sophist-UK/events{/privacy}","received_events_url":"https://api.github.com/users/Sophist-UK/received_events","type":"User","site_admin":false},"body":"I agree. This is not the only place where what unicode.org says is incorrect. (I guess with over 1m code points and 240000 actual characters, there are going to be some errors in the unicode database.)","created_at":"2014-03-28T09:29:52Z","updated_at":"2014-03-28T09:29:52Z","html_url":"https://github.com/musicbrainz/picard/pull/263#discussion_r11059203","pull_request_url":"https://api.github.com/repos/musicbrainz/picard/pulls/263","_links":{"self":{"href":"https://api.github.com/repos/musicbrainz/picard/pulls/comments/11059203"},"html":{"href":"https://github.com/musicbrainz/picard/pull/263#discussion_r11059203"},"pull_request":{"href":"https://api.github.com/repos/musicbrainz/picard/pulls/263"}}}}