Show lines with a single helper function, not one byte at a time

Let's see how hard it is to show UTF-8 characters properly.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Make the 'struct video' contain an array of unicode characters rather than bytes

This is disgusting.  And quite frankly, it's debatable whether this will
ever work.  The "line" structure is still just an array of characters,
so that has to work with utf-8.

But the 'struct video' thing is what represents the actual screen
rectangle, and is fixed-size by the size of the screen.  So making it
contain actual 32-bit unicode characters *may* make sense.

Right now we translate things the same way we always used to, though, so
utf-8 in 'struct line' will not be translated to the proper unicode
array, but to the bytes of the utf-8 representation.  So this really
doesn't improve anything per se yet, just expands the memory use of the
video array.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Show UTF-8 input as UTF-8 output

.. by doing the stupid "convert to unicode value and back" model.

This actually populates the 'struct video' array with the unicode
values, so UTF8 input actually shows correctly.  In particular, the nice
test-file (UTF-8-demo.txt) shows up not as garbage, but as the UTF-8 it
is.

HOWEVER!

Since the *editing* doesn't know about UTF-8, and considers it just a
stream of bytes, the end result is not actually a usable utf-8 editor.
So don't get too excited yet: this is just a partial step to "actually
edit utf8 data"

NOTE NOTE NOTE! If the character buffer contains Latin1, we will
transform that Latin1 to unicode, and then output it as UTF8.  And we
will edit it correctly as the character-by-character data.  Also, we
still do the "UTF8 to Latin1" translation on *input*, so with this
commit we can actually continue to *edit* Latin1 text.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Remove the old utf8_mode thing.

Let's just plan on being fully utf8 some day.  We're not there yet, and
maybe we'll never be, but having the halfway mode is not useful either.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Split up the utf8 helper functions into a file of their own

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Make cursor movement (largely) understand UTF-8 character boundaries

Ok, so it may do odd things if it's not truly utf-8, and when moving up
and down lines that have utf-8 the cursor moves oddly (because the byte
offset within the line stays constant, rather than the character
offset), but with this you can actually open the UTF8 example file and
move around it, and at least some of the movement makes sense.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Use utf8 helper functions for keyboard input

ttgetc() used some homebrew utf8 to unicode translation, limited to just
the normal latin1 characters.  Use the utf8 helper functions to get it
right for the more complex cases.

NOTE! We don't actually handle characters > 0xff right anyway.  And we
still end up doing Latin1 in the buffers on input.  One small step at a
time.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Make kbd macro save area use 'int' instead of short

I'm starting to expand the input value from 'short' (with flags in the
upper eight bytes) to 'int' (with negative values having flags).

Small baby steps.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
utf8: make sure to honor the array length properly

Right now the input side can give partial utf8 input, and that showed
that we didn't properly handle that case.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
character input: make sure we have enough bytes for a full utf8 character

.. but we do have that 0.1s delay, so if somebody feeds us non-utf8
sequences, we won't delay forever.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Expand keycode to 'int' from 'short'

This uses the four high bits for the meta and control key sequences.
This means that we will be limiting our Unicode space to 28 bits, but
that's more than we really need.

It *would* be nicer if we just used the sign bit to mark "we have meta
character information") but that would require bigger changes.  And we
really don't need to worry about 30-bit unicode.  Small steps, remember.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Make 'show_line()' do proper TAB handling

The TAB handling got broken by commit cee00b0efb86 ("Show UTF-8 input as
UTF-8 output") when it stopped doing things one byte at a time.

I'm sure the other special character cases are broken too.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Start actually inserting full utf8 sequences

This makes it possible to cut-and-paste the UTF8 testfile into a new
buffer, and the end result looks correct.

NOTE! We still do various things wrong while editing.  For example,
while the cursor movements were fixed, simple things like deleting a
character still work on single bytes, rather than utf8 characters.

So while this is getting much closer to actually editing UTF-8 data,
it's not there yet.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Start doing character removal properly

This makes actual basic editing work.  Including things like
justify-paragraph etc, so lines get justified by number of UTF8
characters rather than bytes.

There are probably tons of broken stuff left, but this actually seems to
get the basics working right.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Fix vtputc() and simplify show_line by using it again

This re-introduces vtputc() as the way to show characters, which
reinstates the control character handing, and simplifies show_line() in
the process.

vtputc now takes an "int" that is either a unicode character or a signed
char (so negative values in the range [-1, -128] are considered to be
the same as [128, 255]).  This allows us to use it regardless of what
the source of data is.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Fix 'getccol()' and 'getgoal()' functions for multibyte UTF-8 characters

These functions convert the byte offset into the column number
(getccol()) and vice versa (getgoal()).

Getting this right means that moving up and down the text gets us the
right columns, rather than moving randomly left and right when you move
up and down.  We also won't end up in the middle of a utf-8 character,
because we're not just moving into some random byte offset, we're moving
into a proper column.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
uemacs: Remove unused 'lflag' variables from file.c

GCC spotted the following unused variable:

    CC       file.o
  file.c: In function ‘readin’:
  file.c:225:6: warning: variable ‘lflag’ set but not used [-Wunused-but-set-variable]
  file.c: In function ‘ifile’:
  file.c:553:6: warning: variable ‘lflag’ set but not used [-Wunused-but-set-variable]

Signed-off-by: Pekka Enberg <penberg@kernel.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>