utf8-compatibility.py: Improve readability in fill_east_asian_widths()

Alexandre Oliva> fill_east_asian_widths would probably benefit in
Alexandre Oliva> readability and performance from a single regexp that
Alexandre Oliva> could match individual codepoints and ranges:
Alexandre Oliva>
Alexandre Oliva>             match = re.match(
Alexandre Oliva>                 r'^(?P<codepoint1>[0-9A-F]{4,6})(?:\.\.(?P<codepoint2>[0-9A-F]{4,6}))?\s*;\s*(?P<property>[a-zA-Z]+)',
Alexandre Oliva>                 line)
Alexandre Oliva>             if not match:
Alexandre Oliva>               continue
Alexandre Oliva>             start = match.group('codepoint')
Alexandre Oliva>             end = match.group('codepoint2')
Alexandre Oliva>             if not end:
Alexandre Oliva>               end = start
Alexandre Oliva>             for code_point in range(int(start, 16), int(end, 16)+1):
Alexandre Oliva>               east_asian_widths[code_point] = match.group('property')
utf8-compatibility.py: pass file names to check_charmap() and check_width() for less ugly parsing

Alexandre Oliva> instead of reading entire old and new files, how
Alexandre Oliva> about pass the file handle to check_charmap and
Alexandre Oliva> check_width, so that “for line in file:” can be
Alexandre Oliva> used?  Then introduce
Alexandre Oliva> “advance_past_line('CHARMAP')” to position both
Alexandre Oliva> streams after that line.  create_charmap_dictionary
Alexandre Oliva> can then return when it reaches line 'END' or
Alexandre Oliva> 'WIDTH', and neither it nor create_width_dictionary
Alexandre Oliva> would have to skip lines any more.  (removing the
Alexandre Oliva> ugly “start == False”, that had better be spelt
Alexandre Oliva> “not start” :-)
utf8-compatibility.py: More detailed reporting of the CHARMAP changes

Now looks like this:

    ************************************************************
    Report on CHARMAP:
    Total removed characters in newly generated CHARMAP: 0
    Total changed characters in newly generated CHARMAP: 4
    changed: <U9FC0>     /xe9/xbe/x80->/xe9/xbf/x80 CJK Ideograph
    changed: <U9FC1>     /xe9/xbe/x80->/xe9/xbf/x80 CJK Ideograph
    changed: <U9FC2>     /xe9/xbe/x80->/xe9/xbf/x80 CJK Ideograph
    changed: <U9FC3>     /xe9/xbe/x80->/xe9/xbf/x80 CJK Ideograph
    Total added characters in newly generated CHARMAP: 7951
    added: <U037F>     /xcd/xbf GREEK CAPITAL LETTER YOT
    …
    added: <U9FC4>     /xe9/xbf/x80 CJK Ideograph
    added: <U9FC5>     /xe9/xbf/x80 CJK Ideograph
    added: <U9FC6>     /xe9/xbf/x80 CJK Ideograph
    added: <U9FC7>     /xe9/xbf/x80 CJK Ideograph
    added: <U9FC8>     /xe9/xbf/x80 CJK Ideograph
    added: <U9FC9>     /xe9/xbf/x80 CJK Ideograph
    added: <U9FCA>     /xe9/xbf/x80 CJK Ideograph
    added: <U9FCB>     /xe9/xbf/x80 CJK Ideograph
    added: <U9FCC>     /xe9/xbf/x80 CJK Ideograph
    …

Original UTF-8 file:

    …
    <U9F80>..<U9FC3> /xe9/xbe/x80 <CJK>
    <UA000>     /xea/x80/x80 YI SYLLABLE IT
    …

Newly generated UTF-8 file:

    …
    <U9F80>..<U9FBF>     /xe9/xbe/x80 <CJK Ideograph>
    <U9FC0>..<U9FCC>     /xe9/xbf/x80 <CJK Ideograph>
    <UA000>     /xea/x80/x80 YI SYLLABLE IT
    …

The new detailed report above shows clearly that the code points

    changed: <U9FC0>     /xe9/xbe/x80->/xe9/xbf/x80 CJK Ideograph
    changed: <U9FC1>     /xe9/xbe/x80->/xe9/xbf/x80 CJK Ideograph
    changed: <U9FC2>     /xe9/xbe/x80->/xe9/xbf/x80 CJK Ideograph
    changed: <U9FC3>     /xe9/xbe/x80->/xe9/xbf/x80 CJK Ideograph

are not lost but only moved into a different range. In the old file,
the last range contained 68 characters instead of the usual 64.
utf8-compatibility.py: Change detailed report on WIDTH to be more in the style as the new detailed report on CHARMAP

Alexandre Oliva> in check_width, it seems to me that the tests for
Alexandre Oliva> missing and added characters are redundant with the
Alexandre Oliva> change tests, since missing entries have width 1.

Yes, that was redundant. Pravin originally had only the tests
for missing and added and I found it hard to see when a
character was removed with a certain width (and therefore appeared in
“removed”) and was added again with a different width (and therefore
appeared in “added” at a line in the report far away from the line
where it was removed). So it was hard to see when a character changed
width from 1 to 2 or vice versa because it was shown by 2 lines
in the report far away from each other.

So I added the report of the “changed” characters
which actually listed all changes: It listed removing a character
as a change …->1 (because characters not in the file have the default
width 1) and adding a character as a change 1->… .
As this contained all changes, I thought of removing the report
for the added and removed characters, but Pravin liked these
so I left them in.

Now I have changed it again to behave like the new detailed
report for CHARMAP, now “removed”, “added”, and “changed”
are mutually exclusive:

    - removed lists characters which had width 0 or 2 and
      now have 1 because they are not in the file anymore
    - changed lists characters which changed from width 0 to width 2
      or vice versa (i.e. they were in the old file and are in the
      new file but the width has changed)
    - added lists characters which had width 1 because they were
      not in the file but have now with 0 or 2 because they
      have been added.

This seems actually easiest to read.