utf8-gen.py: Use common Python indentation of 4 spaces per indentation level

Style Guide for Python Code:

https://www.python.org/dev/peps/pep-0008/#indentation
utf8-gen.py: Better comment at the beginning of process_charmap()

Alexandre Oliva> some vertical space before the comments that
Alexandre Oliva> introduce process_charmap would be welcome; if the
Alexandre Oliva> comments included a sample input line (like
Alexandre Oliva> process_range), reviewing the code would be easier.
utf8-gen.py: more verbose comment explaining why we skip the reserved code points in EastAsianWidth.txt
utf8-gen.py: Use “with open()” instead of “open() … close()”

I like that style better.
utf8-gen.py: use 2 different functions to write the headers for CHARMAP and WIDTH

Alexandre Oliva> write_comments(flag) is very odd; any reason to not
Alexandre Oliva> write two separate functions, each one outputting the
Alexandre Oliva> text that should precede each portion of the output
Alexandre Oliva> file?
utf8-gen.py: handle commenting out the surrogates in a less confusing way

Alexandre Oliva> the handling of surrogates in process_charmap is
Alexandre Oliva> quite confusing; it appears that in the end the
Alexandre Oliva> script only outputs comments for them; couldn't we
Alexandre Oliva> just skip them in the first place, or use the same
Alexandre Oliva> output logic, but setting a prefix string to an empty
Alexandre Oliva> string or the comment character to it to avoid the
Alexandre Oliva> ugly duplication?  (The prefix string could even be
Alexandre Oliva> modified afterwards to contain the extra zeros
Alexandre Oliva> required for the len(w[0]) != 4 case, like:
Alexandre Oliva>
Alexandre Oliva>     if $is_surrogate:
Alexandre Oliva>       prefix = '%'
Alexandre Oliva>     else
Alexandre Oliva>       prefix = ''
Alexandre Oliva>
Alexandre Oliva>     prefix = prefix + '<U'
Alexandre Oliva>     if len(w) > 4:
Alexandre Oliva>       prefix = prefix + ('0' * (8 - len(w)))
Alexandre Oliva>
Alexandre Oliva>     outfile.write(prefix + ...
utf8-gen.py: Iterate directly over the lines instead of iterating an index in process_charmap()

Alexandre Oliva> as in ctype-compatibility.py, I suggest reworking the
Alexandre Oliva> reading of the input files into using generator
Alexandre Oliva> functions that yield individual (filtered) lines,
Alexandre Oliva> rather than reading entire input files into lists
Alexandre Oliva> that will be iterater over only once.  The iteration
Alexandre Oliva> over the lists with indices, rather than with
Alexandre Oliva> iterators, aggravates the inefficiency.

The whole program takes less then half a second on my machine, I don’t think
we have to worry about efficiency.

The indices were used to peek ahead at the next line if the first line
of a pair like

    3400;<CJK Ideograph Extension A, First>;Lo;0;L;;;;;N;;;;;
    4DB5;<CJK Ideograph Extension A, Last>;Lo;0;L;;;;;N;;;;;

is encountered.

I also think the code with the indices is not so readable and
changed it to iterate over the lines directly and if a “First”
line is encountered, remember it until the next pass through
the loop finds the “Last” line.
utf8-gen.py: Improve readability by removing ', Last' from the name once in process_charmap() instead of doing it many times in process_range()
utf8-gen.py: Improve readability in process_range() by using “'%X' %i” instead of “('%x' %i).upper()”
utf8-gen.py: Improve readability in process_range() by not repeating the same format strings that much
utf8-gen.py: simplify convert_to_hex()

Alexandre Oliva> convert_to_hex is working too hard.  unihex can be
Alexandre Oliva> iterated on, so this should work:
Alexandre Oliva>
Alexandre Oliva>     for c in unihex:
Alexandre Oliva>       hexword = hexword + "/x" + ('%02x' % c)
Alexandre Oliva>
Alexandre Oliva> or even better:
Alexandre Oliva>
Alexandre Oliva>   hexword = ''.join(['/x%02x' % c for c in unihex])
utf8-gen.py: improve convert_to_hex() to convert any integer code point to hex

Alexandre Oliva> how about factoring the inner loop out of
Alexandre Oliva> process_charmaps, so that the logic that converts the
Alexandre Oliva> code point to hexword can be called from
Alexandre Oliva> process_range too?

I changed it so that I can call

    convert_to_hex(i)

for any integer code point i which looks better than

   unihex = chr(i).encode("UTF-8")
   hexword = convert_to_hex(unihex)
utf8-gen.py: improve wording and format of a comment
utf8-gen.py: Fix out-of-date comment at the top of process_width()

Alexandre Oliva> the comments at the top of process_width appear to be
Alexandre Oliva> out of sync with what the function does
utf8-gen.py: Improve formatting and wording of the comment for process_range()
utf8-gen.py: new function ucs_symbol(code_point)

From gen-unicode-ctype.py. Reduces the ugly code duplication
in formatting the output.

Alexandre Oliva> factoring out the logic to turn a code point number
Alexandre Oliva> into a <U...> string would also reduce the amount of
Alexandre Oliva> ugly duplication.  I envision something like this:
Alexandre Oliva>
Alexandre Oliva>   else:
Alexandre Oliva>     iend = int(end, 16)
Alexandre Oliva>     for i in range(int(start, 16), iend, 64):
Alexandre Oliva>       outfile.write(mapline(cpi2Us(i) + ".." + cpi2Us(min(i + 63, iend)),
Alexandre Oliva>                             chr(i), name.split(",")[0]))
Alexandre Oliva>
Alexandre Oliva> def cp2Us(s):
Alexandre Oliva>   return '<U' + s + '>'
Alexandre Oliva> def cpi2Us(i):
Alexandre Oliva>   return cp2Us(('%04X' if i <= 0xffff else '%08X') % i)
Alexandre Oliva> def cps2Us(w):
Alexandre Oliva>   if len(w) > 4:
Alexandre Oliva>     w = '0' * (8 - len(w))
Alexandre Oliva>   return cp2Us(w)
Alexandre Oliva> def mapline(prefix, c, name):
Alexandre Oliva>   unihex = c.encode("UTF-8")
Alexandre Oliva>   hexword = convert_to_hex(unihex)
Alexandre Oliva>
Alexandre Oliva>   return prefix + "     " + hexword + "         " + name + "\n"