Clean up VT52 leftovers from the 256 colours patch

Some references to the unused flag vt52_bold are removed.
Also remove the messing with the 'blink_is_real' flag and similar hacks
that assume overlapping flags that aren't needed with the 256 colour
implementation as we have more than eight stable colours.

Also, the wrap mode should be set to "2" for ATARI compatibility.
Add wcwidth override sequence

ECMA-48: 8.3.90 PEC - PRESENTATION EXPAND OR CONTRACT

CSI Pn SP Z used for character width override.
	Only applies to characters that are not in US-ASCII.
	Does not work with DBCS fonts, only unicode conversion.

    Pn=0   Default
    Pn=1   Force Wide for unicode chars.
    Pn=2   Force narrow for unicode chars.
    Pn=5   Force one Wide unicode char.
    Pn=6   Force one narrow unicode char.

This allows the host to do a DBCS to UTF-8 conversion and keep the
anachronistic alphabetic artifacts of the original encoding.
Add Ansi REP sequence and SCO CSI = g

ECMA-48: 8.3.103 REP - REPEAT
    Notation: (Pn)
    Representation: CSI Pn 06/02
    Parameter default value: Pn = 1

CSI = Pn g
    Display the number Pn as the character from the selected font.
    For 0..255 this is does as the CP437 VGA ROM set, for 256 and
    above it is a unicode character number. Note: SCO ansi does not
    implement values over 255.

This change requires the literal character print routine to be called
from more than one location so that code has been extracted into a
distinct function. Pragmatically this is a long overdue change to help
reduce the size of the huge term_out function.
Fix width override bug.

Width override should ignore combining characters.
Add DIM, CONCEAL, BGBOLD and Blink control.

CSI 2 m	    Set screen colour to dim
CSI 8 m	    Hide the character printed by making the normal display
	    just the background colour. If the text is selected the
	    text is shown
CSI 6 m	    With the advent of 256 colours the SCO Unix bright background
	    sequence can now be implemented using a distinct flag,
	    it doesn't override the 'blink_is_real' flag.

Also in here is a control to allow the text blinking to use any of the
above styles; this includes both the strident PC style and the more
civilised Wyse style.
Add Linux default attribute set.

This is overrides ECMA-48 SDS.
Included so the effects of CSI=F can be reversed.
Make sure all ISO-2022 codes are checked.

They should all be disabled by 'no_remote_charset' some where missed.
Including the vt52 charset modes.
Timer starvation fix

If the remote (or pty for pterm) is giving us data very quickly it can
stop the timers being called. This makes sure we check the timers even
if we're very busy with characters from the host.

NB: Can no longer reproduce this (test machine upgraded).
Fix WM_NETEVENT delayed processing.

Now the function run_toplevel_callbacks() is only called when there
are no pending messages in the Windows queue. This is how it was
designed to work.

Fixes the bug where large fast streams of data from the host will lock
up PuTTY's windows GUI.

But everything still stops if a dialog box or the config box is running
on a different Despatch loop.
Repair handling of WM_NETEVENT (part 2)

This is a simple attempt to reinstate the timer delayed processing of
WM_NETEVENT messages so that processing can continue when secondary
windows (eg the config box) are displayed.

Needs work; seems to miss sometimes.
Fix support of 256 colour palettes.

PuTTY will now redraw it's window when the system palette changes. In
addition the default colours are now at the start of the palette given
to Windows so it's more likely to choose a good colour when the user is
most interested.

The old method using 'UpdateColors' would mostly work on some devices, on
others it would give a completely corrupted set of screen colours. Even
when it did work if you were using XTerm 256 colour mode it would
be unlikely that the colours would ever return to the correct visual
representation and the errors could even spoil the simple 16 colour
collection. The new method simply does a complete redraw when the palette
changes; this is much more robust in the face of dodgy drivers and means
that the colours will always be as correct as possible.

As it now seems stable I've turned it on by default.
Scramble palette

The 6x6x6 cube is scrambled in the palette given to Windows.
This means that even if Windows only gives us part of it we should
get a good selection of colours.
Windows font drawing change.

I've finally given up on using the hDC returned by BeginPaint.
Instead the BeginPaint/EndPaint pair is used to fetch the invalidated
rectangle and tell Windows that it's been validated.

After that we get a proper DC for the entire window and use that for
all the drawing.

This allows us to always update the screen immediatly during the WM_PAINT
message processing and avoid all the bugs in other applications that seem
to trip us when we assume the clip region that BeginPaint sets up is okay.

The end result is that it appears to be much more reliable.

Note, however, we still have the issue that windows is getting slower.
The proportional font update and the do_paint change should help.
Don't let do_paint eat too much CPU
Remove unneeded special case code for proportional fonts.

Proportional fonts used to be too unpredictable to print as strings of
characters because they would frequently spread across multiple cells.
With the change to terminal.c that ensures that text strings will be
completely redrawn when any part is changed this is no longer a problem.

This means that using proportional fonts is now the same speed as fixed
width ones so there's no downside to using a fixed width font that's
stored in a truetype font because it also contains CJK characters.

Prime example is the GNU unifont.

True proportional fonts will now display a little differently, but no
more ugly.
UTF-8 processing update

*) Removed the locking of terminal in UTF-8 mode.
    DOCS standard return always works IF no_remote_charset is off.
*) The alt_utf flag is removed
*) The saved_utf flag is removed.

*) All ISO-2022 controls (except ESC % @) are properly ignored in utf-8 mode.

*) Added rxvt compatible UTF-8 mode for ISO-2022 and UTF-8 at the same time.

*) CSI Pn SP D used to allow host switching to rxvt mixed mode.

*) Under Debian/Ubuntu my locale is en_GB.utf8, not *UTF-8*.
*) Some UTF-8 locales don't have a utf8 suffix eg: en_AG, en_IN, en_NG

Add a config switch to control the new UTF-8 modes on reset.

Option sets the value of the term->utf field on reset.

The configuration values are
    0 => off
	Terminal uses the configured codepage after a reset.
    1 => on
	Terminal uses UTF-8 after a reset, DOCS RETURN is used to get to
	the configured codepage.
    2 => mixed
	Terminal uses UTF-8 after a reset. But if the character sequence
	isn't legal UTF-8 it will be treated as the configured codepage.

Rather than overriding the line character set, the UTF8 locale override
forces this flag to UTF-8. This means that the 8bit codepage is still
available if the host explicitly requests it (ISO8859-1 if the line
codepage is UTF-8)

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Reasoning for UTF-8 change ...

No version of curses understands ISO-2022 (possibly outside Japan)
but some do understand UTF-8. The Japaneese version of ISO-2022 is
more complex.

The utf flag is not controlled by most host software, only specialist
software (and scripts) that understand ISO-2022 and UTF-8 will use
it. Therefor the existance of multi copies of the flag (alt & saved)
is not useful and has caused confusion.

This is because the UTF-8 functionality of most host software is
controlled through the 'locale' functions. These have no built in
facilities for sending initilisation sequences to a terminal to change
it's mode. However, a host script can change the locale and so should
be able to change the terminal to match.

    if [ "`locale charmap 2>/dev/null`" = "UTF-8" ]
    then echo -ne '\033%G'		# DOCS UTF-8
    else echo -ne '\033%@'		# DOCS RETURN
    fi

Most utf-8 software expects normal sequences to have no effect on a
UTF-8 terminal, ncurses for example may still send smacs strings even
when using UTF-8 line drawing characters.

For the future it's likely that the terminal will run mostly in UTF-8
so this mode will become even less useful.

=-=
The automatic locking of the utf flag for UTF-8 fonts is confusing in
that it means there are two distinct versions of PuTTY control sequences
(with or without ISO-2022) without an obvious configuration warning. In
addition for current versions of PuTTY the functionality is provided by
the option to disable remote control of character sets.

=-=
When the 'DOCS return' sequence is used the ISO-2022 engine must be
returned to exactly the state it was in when the original 'DOCS' was
received. (ECMA-035)

=-=
The rxvt compatibility (wishlist/utf8-plus-vt100) means that all the
ISO-2022 sequences can be made to continue to work while in UTF-8
mode. This is only reasonable when you have a hellspawn host that has
an unlabeled mixture of unicode and non-unicode applications and data
but it can still be useful.  (eg: if parts of your filesystem have been
around for nearly 20 years like mine) Even then there are reasonable
arguments that this should be done by translation applications not an
emulator. For my usage this is a 'normally off' option that can be turned
on to 'get the ****ing thing working'.

The pure UTF-8 mode is still the default. Note: As the escape sequence to
get into the mixed mode is NOT an ISO-2022 sequence the rules about DOCS
restoring the ISO-2022 state do not apply.

This mixed mode also means that the UTF-8 decoder had to be changed;
instead of mostly using an FSM with a single character lookahead buffer
the UTF-8 decoder has been changed to use only a lookahead buffer.

This buffer is also used instead of the tiny FSM for the printer output
and is likely to be a good solution for any other emulation extensions
including full ISO-2022, Japanese auto-detection of character encoding,
other complex encodings like the Chinese standard and even full emulation
replacements like a hardline VT220 or Wyse.
Add comment about UTF-8 locale detection.
Add new interpretation for XTerm SGR 38/48

After actually seeing a copy of ITU T.416-199303 XTerm was modified
to be able to use the character ':' as the parameter separator for 256
colour mode.

Unfortunatly, this documentation is still very unclear and the section
looks like it may have been written without fully understanding the
confused phrasing of ECMA-48.

As this is the only sequence that officially uses ':', is likely to be
the only one that ever uses that character and XTerm in effect does it
too. I've simply simply made ':' an alias of ';'.

My understanding of the combined documentation is that the sequences should
be of the form:

    CSI 48:5:1 ; 38:5:7 m

This is due to my reading the phrase "parameter sub-string" (in T.416)
as referring to a sub-string attached to a "parameter value". Not a
reference to the string version of the "parameter value" before it is
formally defined to be a decimal integer as it is in ECMA-48.

However, this is not really relevent as for compatibility reasons all
variations will have to be supported.
Add XTerm/Konsole truecolour mode.

Phase 1, map the colours to the nearest XTerm 256 colour mode colour.
This is how XTerm does it.
Adding better support for Windows MBCS (codepages)

Adding better support for some 国家标准 character sets. (ie Chinese standard
character sets), Japanese, Korean and Thai.  Including the Shift-JIS variants.

The characters are displayed at the expected widths.

This does NOT need a specific font as long as the unicode system can find the
requsite characters.

Note the change in the defaults; it is now recommended to explicitly
specify the line character set for DBCS systems.

Also added these Windows charsets to the list.

Unlike DBCS fonts this can be made to work for the Unix port as it only
requires Codepage to Unicode character and width tables which already
exist in the iconv() library.