Pango font performance update

This change improves the speed of the best case pango font
processing.  It removes multiple calls to pango_layout_set_text and
pango_layout_get_pixel_extents that were used to check that every prefix
string of a given draw_text is actually the correct size.

Instead the function now checks just once before painting the text
that the size of the whole region is correct. If it's not correct it
falls back to painting one character at a time immediatly rather than
optimistically trying lots of substrings; this normally improves the
draw time of complex screens and proportional fonts too.

If there are compensating layout issues within the string they won't
cause problems because terminal.c already has code to redraw any row
of text that was drawn as one operation if it's split.
BUG: Wide characters and the passive cursor being continually redrawn.

Updated the dirtyrect processing so it deals properly with DBCS fonts in
direct to font mode;

  It'll redraw the whole rectangle if two side by side become the same
  as they do when the cursor moves across them.

  The UCSWIDE character wasn't being put into the display array properly
  causing repeated redraws.

  The passive cursor was being redrawn every blink; but it doesn't blink.

This bug needed a small refactor too see WTH was happening, I think the
new version is a lot clearer myself and most of the diffs are whitespace.
Restore reading from host when menu displayed.

If menu or configuration box is displayed PuTTY no longer accepts
characters from the host like it used to.

This fixes that by adding another timer that pulls data from the network
even when the main windows loop is not being run.

: > Do you happen to know when it 'used to'? Any idea what broke this?
: I put the effect in back when I revamped the message processing on
: Windows so that the host couldn't lock PuTTY up by sending characters
: too fast for the local machine to process. The processing of characters
: from the host was moved to the main message loop so that it would be the
: lowest priority task, the problem was that when the config dialog was
: up it used it's own local message loop, so I added processing on the
: main timer tick that would both allow processing to continue in this
: case and allow some characters to be processed even if some other set
: of messages was taking up all the processeing time.
:
: I guess it went away when the timer stuff was rewritten for the unix port.
Windows font drawing performace upgrade.

Two issues here.

Firstly. Windows is getting slower at drawing characters to the screen,
so much so that sometimes it overruns the time limits and eats into
the time that we should be using to process characters received from
the host. So add a change to make do_paint not accept a redraw request
until there has been 'enough' time since the last one.

In addition the WM_PAINT message has been being used to directly draw
characters to the screen. Because of issues with some machines this can
cause screen corruption (white lines).

Note: not all the corruption is cured by this patch as it stands; white
marks can still occur in the border area and the scroll bar.

With Windows 7 this has become mostly irrelevent as, usually, windows
will now 'save-unders' for the transparency.

Tested on W95 OSR2, W2k SP4, WXP SP3, WXP64 SP2, Win7 SP1.
Timer starving fix

If the remote (or pty for pterm) is giving us data very quickly it can
stop the timers being called. This makes sure we check the timers even
if we're very busy with characters from the host.
Minimise, maximise, etc are as annoying as other terminal resize operations.

Note they ARE resize operations so treat them as such and disable them
if requested.
Patch for vttest width changing test.

Most applications realise that terminal resizes are requests and that
you likely won't get what you ask for.  Programs that use the sequence
DECCOLM aren't so smart.
Adjust DEC-MCS character set.

U+2426 "␦" is defined as the substitution character in DEC-MCS

Note: dec_mcs is a 94 character set so 0xA0 and 0xFF are not defined in
a different way when it's mapped into GR.
Changes to the internal usage of unicode areas for character set conversion.

The unix port is using the U+D800 range for the same jobs as the windows
port uses the U+F100 range. Set it to use the same range of characters.

Note: unlike windows this is ONLY used if the font doesn't have a
recognised character set so this change doesn't make the DTF range
compatible with the Linux console.

In a similar issue terminal.c is using the CSET_* codes for two different
tasks; the screen map characters and the line characters. Making the
distinction explicit should clarify things.

We now have a 32bit space for the characters so the hack of placing
the CSET_* code overlapping the surrogates is no longer required. We
can instead place them past the end of the valid unicode characters
(ie U+180000). Note I've chosen 0x180000 rather than 0x200000 to preserve
the compression of the scrollback.

Unfortunatly I can't do the same for the DTF range as it wouldn't work
for the UTF-16 platform. I can, however, move it to the "Low Surrogates"
area where it shouldn't interfere with the normal use of surrogates.

Added Linux DTF range (U+F000 -> CP437) explicitly into the emulation
because we no longer have the "happy coincidence" of the OEMCP being
mapped to U+F000. This means it could now be disabled for pure utf8 mode.
Colour scheme controls.

This is possibly related to: /wishlist/colour-scheme.html

These colour schemes are not stored in the session file by have to be
dynamically configured from the host.

Possibly GUI option to save current colour set to saved session needed ?
This is a fix for the direct to font DBCS cursor.

If we're sitting on a DBCS lead byte it extends the cursor to two
characters rather then just one.
UTF-8 processing update

1) Removed the locking of terminal in UTF-8 mode on font selection.
2) The alt_utf flag is removed
3) The saved_utf flag is removed.
4) On terminal reset the utf flag is set to:
	(term->ucsdata->line_codepage==CP_UTF8)
5) All ISO-2022 controls (except ESC % @) are properly ignored in utf-8 mode.

6) Added rxvt compatible UTF-8 mode for ISO-2022 and UTF-8 at the same time.

Also added.
7) Under Debian/Ubuntu my locale is en_GB.utf8, not *UTF-8*.
8) Some UTF-8 locales don't have a utf8 suffix eg: en_AG, en_IN, en_NG
9) Stop the BMP repeating for 16bit UCS fonts (Win32 & X11, not Pango)
10) High bit characters for title an icon are translated to utf-8.
11) CSI Pn SP Z used for character width override.
12) CSI Pn SP D used for current codepage changing.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Reasoning for UTF-8 change ...

No version of curses understands ISO-2022 (outside Japan) but some do
understand UTF-8. The Japaneese version of ISO-2022 is more complex.

The utf flag is not controlled by most host software, only specialist
software (and scripts) that understand ISO-2022 and UTF-8 will use
it. Therefor the existance of multi copies of the flag (alt & saved)
is not useful and has caused confusion.

This is because the UTF-8 functionality of most host software is
controlled through the 'locale' functions. These have no built in
facilities for sending initilisation sequences to a terminal to change
it's mode. However, a host script can change the locale and so should
be able to change the terminal to match.

    if [ "`locale charmap 2>/dev/null`" = "UTF-8" ]
    then echo -ne '\033%G'		# DOCS UTF-8
    else echo -ne '\033%@'		# DOCS RETURN
    fi

Alternativly ...
    if [ "`locale charmap 2>/dev/null`" = "UTF-8" ]
    then echo -ne '\033[0;1400 D'	# UTF-8 without ISO-2022
    else echo -ne '\033[0;2022 D'	# Normal ECMA-35 (ISO-2022)
    fi

Most utf-8 software expects normal sequences to have no effect on a
UTF-8 terminal, ncurses for example may still send smacs strings even
when using UTF-8 line drawing characters.

For the future it's likely that the terminal will run mostly in UTF-8
so this mode will become even less useful.

=-=
The automatic locking of the utf flag for UTF-8 fonts is confusing in
that it means there are two distinct versions of PuTTY control sequences
(with or without ISO-2022) without an obvious configuration warning. In
addition for current versions of PuTTY the functionality is provided by
the option to disable remote control of character sets.

=-=
When the 'DOCS return' sequence is used the ISO-2022 engine must be
returned to exactly the state it was in when the original 'DOCS' was
received. (ECMA-035)

=-=
The rxvt compatibility (wishlist/utf8-plus-vt100) means that all the
ISO-2022 sequences can be made to continue to work while in UTF-8
mode. This is only reasonable when you have a hellspawn host that has
an unlabeled mixture of unicode and non-unicode applications and data
but it can still be useful.  (eg: if parts of your filesystem have been
around for nearly 20 years like mine) Even then there are reasonable
arguments that this should be done by translation applications not an
emulator.

This mixed mode also means that the UTF-8 decoder had to be changed;
instead of mostly using an FSM with a single character lookahead buffer
the UTF-8 decoder has been changed to use only a lookahead buffer.
This buffer is also used instead of the tiny FSM for the printer output
and is likely to be a good solution for full ISO-2022.

The unmixed mode is still the default. Note: As the escape sequence to
get into the mixed mode is NOT an ISO-2022 sequence the rules about DOCS
restoring the ISO-2022 state do not apply.

=-=
ECMA-48: 8.3.90 PEC - PRESENTATION EXPAND OR CONTRACT

CSI Pn SP Z used for character width override. Only applies to UTF-8
		characters that are not in US-ASCII.
	Pn=0   Default
	Pn=1   Force Wide for unicode chars.
	Pn=2   Force narrow for unicode chars.
	Pn=5   Force one Wide unicode char.
	Pn=6   Force one narrow unicode char.

ECMA-48: 8.3.53 FNT - FONT SELECTION

CSI Pf ; Pc; SP D
	Change current codepage.
	Pf must be zero or default.
	Pc is a supported codepage number

	2022        Switch to ISO 2022 mode.
	65001       UTF-8 in normal DOCS mode.
	4110        UTF-8 randomly mixed with ISO 2022.
	1400        UTF-8 with ISO-2022, ECMA-35 and DOCS RETURN disabled.

ESC % G   DOCS UTF-8
	Switch to UTF-8 all ISO-2022 sequences except ESC % @ are disabled.
ESC % @	  DOCS RETURN
	Switch back to iso 2022 mode.
Add a config switch to control the new UTF-8 modes.

Option sets the value of the term->utf field on reset.

The configuration values are
    0 => off
	    Terminal uses the configured codepage after a reset.
    1 => on
	    Terminal uses UTF-8 after a reset, DOCS RETURN is used to get to
	    the configured codepage.
    2 => mixed
	    Terminal uses UTF-8 after a reset. But if the character sequence
	    isn't legal UTF-8 it will be treated as the configured codepage.
    3 => locked
	    This isn't in the configuration, but if added it'll be UTF-8
	    without DOCS RETURN enabled. The non-ISO-2022 switching sequence
	    CSI Pf ; Pc; SP D will still work.
Adding better support for some Chinese character sets

Adding better support for some 国家标准 character sets. (ie Chinese standard
character sets)

This does NOT need a specific font as long as the unicode system can find the
requsite characters.

Note the change in the defaults; it is now reccomended to explicitly
specify the line character set for DBCS systems.

Also added a few Windows charsets and moved UTF-8 right to the top of
the list.

Minor change to ldiscucs.c, for non-UTF16 machines the characters
for surrogate pairs have already been checked.
This patch adds ANSI.SYS, SCO Xenix 2.3.2 and a general ANSI emulation.

The SCO Xenix is a well defined but rather old style emulation similar
in many respects to the ANSI.SYS emulation but a bit nicer. The later
SCO Unix variants are far more uncertain as they cover may versions of
SCO Unix where the emulation was changed to maintain or provide minimal
compatibility with other systems.

In the SCO Xenix emulation I have assumed the mapchan(1) facility has
been used to set the default character set to the putty line codepage. So
if your application assumes the mapchan process hasn't been run you will
need to set the line charset to CP437. The ANSI.SYS emulation forces CP437.

echo -e '\e[4040"p'   # Set normal PuTTY (xterm) mode.
echo -e '\e[4041"p'   # Set modern Ansi mode, all available non VT sequences.
echo -e '\e[4042"p'   # Set ANSI.SYS emulation
echo -e '\e[4043"p'   # Set SCO Xenix

echo -e '\e[4030"p'   # Set pure VT52 mode
echo -e '\e[4031"p'   # Set pure VT220+ mode

These and all the compatibility stuff can be turned off using the compile
option DISABLE_COMPAT. Note, CSI 4030"p turns off _nearly_ all the CSI
sequences, the exception being ones that begin "ESC [ 404".
The SCO emulation overrides the configured function key definitions.

The SCO Xenix emulation does not have the reverse or graphic attribute
variables. However, I do have full details if these are needed.

Attributes DIM and CONCEAL have been added. Concealed text is shown by
highlighting it with the mouse.

Altered CSI 21m to be dup of CSI 22m; I didn't know about this convention
when I chose to use single underline as a near enough equlivent to
double underline.

The SCO ACS character set on unix has been fixed; added the OEM glyphs
for characters 1..31.
The default SCO mapping for the number pad '5' key has been fixed.
The ECMA-48 'REP' code, repeat character has been added.
I've added SCO's new CSI 13m mode and the CSI=g code.
Removed the supposedly SCO CSI 6m.
Linux CSI 8 ] for default colours. CSI 8 ; 1 ] to reset.
Added CSI 8 ; 2 ] to reconfigure the default colours and the BLINK
attribute to blink any other attribute.

In ANSI.SYS mode CSI-J homes cursor.

Added XTerm's vttest modes; CSI ?40 l/h & CSI ?45 l/h
    This means DECCOLM is disabled by default and stops xterm's "tput init"
    clearing the screen.

Default blink now to DIM not blank, set with:
   echo -e '\e[0;2m\e[8;2]\e[m'
Set back to PC Style with:
   echo -e '\e[0;8m\e[8;2]\e[m'

I haven't added GUI configs for any of this; don't know which you'd prefer.
    Possibilities:
	Terminal Type: xterm/ansi/ansisys/scoansi/vt220/vt52
	Blink attribute: Dim/Conceal/Reverse ...

I've had to move the literal character display into it's own function
so I can call it from multiple places.

Readded Linux DTF range (U+F000 -> CP437) explicitly into the emulation.
This means I no longer need the "happy coincidence" of the OEMCP being
mapped to U+F000.

Cleaned out "vt52_bold" variable; it does nothing.