Update “steps-to-generate-ctype” to explain how to use gen-unicode-ctype.py
ctype-compatibility.py: Make get_lines_from_file() work even if the files ends with a continuation line

Alexandre Oliva> get_lines_from_file doesn't check that current_line
Alexandre Oliva> is empty at the end of the loop.  that's fine,
Alexandre Oliva> because it would indicate a file terminated with a
Alexandre Oliva> continuation line.  I'm not sure this is worth
Alexandre Oliva> checking and reporting an error, but maybe a comment
Alexandre Oliva> would be in order
ctype-compatibility.py: make regexp in extract_character_classes_and_code_points() more strict

Alexandre Oliva> I'm a little bit concerned with the regexp in
Alexandre Oliva> extract_character_classes_and_code_points.  I
Alexandre Oliva> recommend using re.escape(char_class) instead of
Alexandre Oliva> plain char_class, so that, if the char_classes ever
Alexandre Oliva> contains rx-active characters, we won't get
Alexandre Oliva> unintended matches or rx compilation failures.
Alexandre Oliva> Furthermore, the current rx could match such
Alexandre Oliva> extraneous line prefixes as «class "upper » or
Alexandre Oliva> «lower"; ».  Maybe drop the '?'s from the current rx
Alexandre Oliva> and accept '|'+re.escape(char_class)+'\s+' as an
Alexandre Oliva> *alternative* to the partiall rx between '^' and
Alexandre Oliva> '\s+', or check that if any of the before- and
Alexandre Oliva> after-char_class groups is empty, then so is the
Alexandre Oliva> other, reporting an error otherwise.  (Am I being too
Alexandre Oliva> paranoid?)

I don’t think this can really happen and it makes the regexp
much more ugly.
ctype-compatibility.py: exit with an error in process_chars() if none of the regexps matches

Alexandre Oliva> process_chars has no error reporting if none of the
Alexandre Oliva> regexps match,

This should not happen (and does not happen at the moment) but
maybe it is better to check just in case.

Alexandre Oliva> and it could be reworked to use a single regexp to
Alexandre Oliva> cover all cases, like:
Alexandre Oliva> '^<U(?P<codepoint>...)>(?:(?:\.\.(?:\(2\)\.\.)?)<U(?P<codepoint2>...)>)$'
Alexandre Oliva> +
Alexandre Oliva> '|^\(<U(?P<codepointf>...)>,<U(?P<codepointm>...)>)\)$'
Alexandre Oliva> and then testing the match.group()s to tell the cases
Alexandre Oliva> apart.  Or maybe have two separate regexps, but
Alexandre Oliva> breaking up the handling of classes and of maps into
Alexandre Oliva> separate functions, so that you don't have to
Alexandre Oliva> repeatedly test for one case or the other.  It would
Alexandre Oliva> also make the code far more readable: it took me a
Alexandre Oliva> while to realize why the comma separator was handled
Alexandre Oliva> so differently from the two other kinds of
Alexandre Oliva> separators, before I finally hit the parentheses in
Alexandre Oliva> the regexp ;-)

Hm, for me it seems readable, I don’t think a single
regexp would really improve readability.
ctype-compatibility.py: don’t expand ranges in cpcheck

Alexandre Oliva> cpcheck needlessly expands ranges; I think this could
Alexandre Oliva> be avoided by rearranging the loop so as to use
Alexandre Oliva> ranges as generators, rather than expanding them,
Alexandre Oliva> like (untested):
Alexandre Oliva>
Alexandre Oliva>     for r in code_point_list_with_ranges:
Alexandre Oliva>       for code_point in [r] if type(r) == type(int()) else range(r[0], r[1]+1):
Alexandre Oliva>         for c in char_classes:

I like that because it is shorter and seems more readable to me.
ctype-compatibility.py: fix some typos in strings

Alexandre Oliva> there are a couple of “charcter” typos in tests
ctype-compatibility.py: make conversion from ctype_dict to ctype_dict2 more efficient

Alexandre Oliva> the conversion of ctype_dict to ctype_dict2 tests
Alexandre Oliva> whether key is in a list; I *think* testing the type
Alexandre Oliva> of value might be more efficient, while avoiding
Alexandre Oliva> hard-coding the list of maps.  Also, pulling this
Alexandre Oliva> test out of the loop can avoid repeating the test for
Alexandre Oliva> every loop iteration, even if duplicating the inner
Alexandre Oliva> loop in the if and else cases.

Readability suffers a bit but it is faster, 5.4 seconds
instead of 6.2 seconds for the whole script on my machine.
ctype-compatibility.py: Add a function to increase the number of errors and print a message

Alexandre Oliva> tests (and cpcheck) have a large number of code
Alexandre Oliva> blocks that go:
Alexandre Oliva>
Alexandre Oliva>     if (...):
Alexandre Oliva>       number_of_errors += 1
Alexandre Oliva>       print(...%{...})
Alexandre Oliva>
Alexandre Oliva> it would be nice and less error prone to factor out
Alexandre Oliva> at least part of this pattern.  For example, define a
Alexandre Oliva> function that increments number_of_errors and prints
Alexandre Oliva> the given string, so that this could be rewritten as:
Alexandre Oliva>
Alexandre Oliva>     if (...):
Alexandre Oliva>       cperror(...%{...})
Alexandre Oliva>
Alexandre Oliva> it might be even nicer if the expression, the format
Alexandre Oliva> string and the bindings dictionary could be turned
Alexandre Oliva> into a lambda or somesuch in a list of tests, but I
Alexandre Oliva> can't think of a clean way to do that without
Alexandre Oliva> needlessly applying the format operator when the test
Alexandre Oliva> passes.  Something to think about...