Windows codepages do not have C1 controls

MS CP125X do NOT actually have any high control characters, the MS tables
show some of them but they are never interpreted by Microsoft software
and, as they have done before, MS will replace them by any new characters
that may be useful in the future.

Also eliminates as possible security risk where control characters are
not expected in Windows csets by host software.
Fix the scoasc translation on unix

There should be a set of VGA ROM characters in the first 32 positions
of the character set to match with Linux, scoansi and Windows PuTTY.
Function x11font_draw_text uses the UCS2 character set

Bugfix to stop the character set being repeated above U+10000
SCO Ansi Keyboard correction

The centre key (the '5') of the number pad should generate a CSI E for SCO.
Correct Window title character set on windows.

The characters are converted to unicode before use.
Changes to the internal usage of unicode areas for character set conversion.

The unix port is using the U+D800 range for the same jobs as the windows
port uses the U+F100 range. Set it to use the same range of characters.

Note: unlike windows this is ONLY used if the font doesn't have a
recognised character set so this change doesn't make the DTF range
compatible with the Linux console.

In a similar issue terminal.c is using the CSET_* codes for two different
tasks; the screen map characters and the line characters. Making the
distinction explicit should clarify things.

We now have a 32bit space for the characters so the hack of placing
the CSET_* code overlapping the surrogates is no longer required. We
can instead place them past the end of the valid unicode characters
(ie U+180000). Note I've chosen 0x180000 rather than 0x200000 to preserve
the compression of the scrollback.

Unfortunatly I can't do the same for the DTF range as it wouldn't work
for the UTF-16 platform. I can, however, move it to the "Low Surrogates"
area where it shouldn't interfere with the normal use of surrogates.

Added Linux DTF range (U+F000 -> CP437) explicitly into the emulation
because we no longer have the "happy coincidence" of the OEMCP being
mapped to U+F000. This means it could now be disabled for pure utf8 mode.
Clean up VT52 leftovers from the 256 colours patch

Some references to the unused flag vt52_bold are removed.
Also remove the messing with the 'blink_is_real' flag that isn't
needed with the 256 colour implementation as we have more than
eight stable background colours.

Also, the wrap mode should be set to "2" for ATARI compatibility.
Add wcwidth override sequence

ECMA-48: 8.3.90 PEC - PRESENTATION EXPAND OR CONTRACT

CSI Pn SP Z used for character width override.
	Only applies to characters that are not in US-ASCII.
	Does not work with DBCS fonts, only unicode conversion.

    Pn=0   Default
    Pn=1   Force Wide for unicode chars.
    Pn=2   Force narrow for unicode chars.
    Pn=5   Force one Wide unicode char.
    Pn=6   Force one narrow unicode char.

This allows the host to do a DBCS to UTF-8 conversion and keep the
anachronistic alphabetic artifacts of the original encoding.
Add Ansi REP sequence and SCO CSI = g

ECMA-48: 8.3.103 REP - REPEAT
    Notation: (Pn)
    Representation: CSI Pn 06/02
    Parameter default value: Pn = 1

CSI = Pn g
    Display the number Pn as the character from the selected font.
    For 0..255 this is does as the CP437 VGA ROM set, for 256 and
    above it is a unicode character number. Note: SCO ansi does not
    implement values over 255.

This change requires the literal character print routine to be called
from more than one location so that code has been extracted into a
distinct function. Pragmatically this is a long overdue change to help
reduce the size of the huge term_out function.
Fix width override bug.

Width override should ignore combining characters.
Add DIM, CONCEAL and BGBOLD

CSI 2 m	    Set screen colour to dim
CSI 8 m	    Hide the character printed by making the normal display
	    just the background colour. If the text is selected the
	    text is shown
CSI 6 m	    With the advent of 256 colours the SCO Unix bright background
	    sequence can now be implemented using a distinct flag,
	    it doesn't override the 'blink_is_real' flag.

The text blinking can be set to use various styles, including the strident
PC styles and the more civilised Wyse style.
Add Linux default attribute set.

This is overrides ECMA-48 SDS.
Included so the effects of CSI=F can be reversed.
Make sure all ISO-2022 codes are checked.

They should all be disabled by 'no_remote_charset' some where missed.
Including the vt52 charset modes.
Timer starvation fix

If the remote (or pty for pterm) is giving us data very quickly it can
stop the timers being called. This makes sure we check the timers even
if we're very busy with characters from the host.

NB: Can no longer reproduce this (test machine upgraded).
UTF-8 processing update

*) Removed the locking of terminal in UTF-8 mode.
    DOCS standard return always works IF no_remote_charset is off.
*) The alt_utf flag is removed
*) The saved_utf flag is removed.

*) All ISO-2022 controls (except ESC % @) are properly ignored in utf-8 mode.

*) Added rxvt compatible UTF-8 mode for ISO-2022 and UTF-8 at the same time.

*) CSI Pn SP D used to allow host switching to rxvt mixed mode.

*) Under Debian/Ubuntu my locale is en_GB.utf8, not *UTF-8*.
*) Some UTF-8 locales don't have a utf8 suffix eg: en_AG, en_IN, en_NG

Add a config switch to control the new UTF-8 modes on reset.

Option sets the value of the term->utf field on reset.

The configuration values are
    0 => off
	Terminal uses the configured codepage after a reset.
    1 => on
	Terminal uses UTF-8 after a reset, DOCS RETURN is used to get to
	the configured codepage.
    2 => mixed
	Terminal uses UTF-8 after a reset. But if the character sequence
	isn't legal UTF-8 it will be treated as the configured codepage.

Rather than overriding the line character set, the UTF8 locale override
forces this flag to UTF-8. This means that the 8bit codepage is still
available if the host explicitly requests it (ISO8859-1 if the line
codepage is UTF-8)

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Reasoning for UTF-8 change ...

No version of curses understands ISO-2022 (possibly outside Japan)
but some do understand UTF-8. The Japaneese version of ISO-2022 is
more complex.

The utf flag is not controlled by most host software, only specialist
software (and scripts) that understand ISO-2022 and UTF-8 will use
it. Therefor the existance of multi copies of the flag (alt & saved)
is not useful and has caused confusion.

This is because the UTF-8 functionality of most host software is
controlled through the 'locale' functions. These have no built in
facilities for sending initilisation sequences to a terminal to change
it's mode. However, a host script can change the locale and so should
be able to change the terminal to match.

    if [ "`locale charmap 2>/dev/null`" = "UTF-8" ]
    then echo -ne '\033%G'		# DOCS UTF-8
    else echo -ne '\033%@'		# DOCS RETURN
    fi

Most utf-8 software expects normal sequences to have no effect on a
UTF-8 terminal, ncurses for example may still send smacs strings even
when using UTF-8 line drawing characters.

For the future it's likely that the terminal will run mostly in UTF-8
so this mode will become even less useful.

=-=
The automatic locking of the utf flag for UTF-8 fonts is confusing in
that it means there are two distinct versions of PuTTY control sequences
(with or without ISO-2022) without an obvious configuration warning. In
addition for current versions of PuTTY the functionality is provided by
the option to disable remote control of character sets.

=-=
When the 'DOCS return' sequence is used the ISO-2022 engine must be
returned to exactly the state it was in when the original 'DOCS' was
received. (ECMA-035)

=-=
The rxvt compatibility (wishlist/utf8-plus-vt100) means that all the
ISO-2022 sequences can be made to continue to work while in UTF-8
mode. This is only reasonable when you have a hellspawn host that has
an unlabeled mixture of unicode and non-unicode applications and data
but it can still be useful.  (eg: if parts of your filesystem have been
around for nearly 20 years like mine) Even then there are reasonable
arguments that this should be done by translation applications not an
emulator. For my usage this is a 'normally off' option that can be turned
on to 'get the ****ing thing working'.

The pure UTF-8 mode is still the default. Note: As the escape sequence to
get into the mixed mode is NOT an ISO-2022 sequence the rules about DOCS
restoring the ISO-2022 state do not apply.

This mixed mode also means that the UTF-8 decoder had to be changed;
instead of mostly using an FSM with a single character lookahead buffer
the UTF-8 decoder has been changed to use only a lookahead buffer.

This buffer is also used instead of the tiny FSM for the printer output
and is likely to be a good solution for any other emulation extensions
including full ISO-2022, Japanese auto-detection of character encoding,
other complex encodings like the Chinese standard and even full emulation
replacements like a hardline VT220 or Wyse.
Add comment about UTF-8 locale detection.
Add new interpretation for XTerm SGR 38/48

After actually seeing a copy of ITU T.416-199303 XTerm was modified
to be able to use the character ':' as the parameter separator for 256
colour mode.

Unfortunatly, this documentation is still very unclear and the section
looks like it may have been written without fully understanding the
confused phrasing of ECMA-48.

As this is the only sequence that officially uses ':', is likely to be
the only one that ever uses that character and XTerm in effect does it
too. I've simply simply made ':' an alias of ';'.

My understanding of the combined documentation is that the sequences should
be of the form:

    CSI 48:5:1 ; 38:5:7 m

This is due to my reading the phrase "parameter sub-string" (in T.416)
as referring to a sub-string attached to a "parameter value". Not a
reference to the string version of the "parameter value" before it is
formally defined to be a decimal integer as it is in ECMA-48.

However, this is not really relevent as for compatibility reasons all
variations will have to be supported.
Add XTerm/Konsole truecolour mode.

Phase 1, map the colours to the nearest XTerm 256 colour mode colour.
This is how XTerm does it.
Adding better support for Windows MBCS (codepages)

Adding better support for some 国家标准 character sets. (ie Chinese standard
character sets), Japanese, Korean and Thai.  Including the Shift-JIS variants.

The characters are displayed at the expected widths.

This does NOT need a specific font as long as the unicode system can find the
requsite characters.

Note the change in the defaults; it is now recommended to explicitly
specify the line character set for DBCS systems.

Also added these Windows charsets to the list.

Unlike DBCS fonts this can be made to work for the Unix port as it only
requires Codepage to Unicode character and width tables which already
exist in the iconv() library.
Waaay overkill on uuid generation

The method used to generated the stable GUID values for Windows is a
variant of the type 3 and type 5 UUID generation methods.

For these you only need one properly generated UUID to make a namespace.

I have left it advertised as a type 4 UUID because it doesn't follow the
exact procedure (or use the correct cryptographical hash) for either of
the standard ones.