CleverRaven/Cataclysm-DDA — pull request #8041 by BevapDin (2014-06-23 13:54:59+00:00)
Title: Support for input in Unicode [CR]

In memory of: ¥Ö£€$¢Ä₡₢₣₤₥ä₦₧₨µ₩₪₫₭₮₯₹ (that's an example Unicode string)

It seems that displaying of Unicode is already fully functional? Therefor this handles only the input. Not all fonts support this (you might have to change it in config/fonts.json). I use "DejaVu Sans Mono Book".

Tested on my system which has only a multi-byte ncurses library. I'm not sure how this behaves on other systems. This is not tested with multi-cell characters (for which `mk_wcwidth` would return something greater than 1).

----

This PR includes a new class, `utf8_wrapper` which is a simple wrapper of an UTF-8 encoded `std::string`. It uses the already existing function `UTF8_getch` and `mk_wcwidth`. One can add, remove, insert etc. like with a normal `std::string` - but the class makes sure that multi-byte characters are handled as a single character. It also reports only the expected on-screen length / the number of Unicode characters, not the amount of bytes it contains.

For a string that contains only a single Unicode character, which is expressed as 3 bytes in UTF-8, the class returns a length of 1, while the actual string data contains 3 bytes. This allows the input dialog to navigate in those strings without destroying it and without caring about it being UTF-8. This also means functions like `substr` or `erase` or `insert` operate on the UTF-8 characters, not on the bytes. For example `insert(3, other)` will insert the other string behind the 3. UTF-8 character, which might be behind the 3. byte.

`getch` returns a single byte, when the user enters a Unicode character it is translated into a UTF-8 encoded string (done by the console). Each byte of that sequence is returned by repeatedly calling `getch`. If the Unicode character is encoded as three bytes, one has to call `getch` three times to get the full UTF-8 sequence. The function `insert_from_getch` does this for you. It inserts the UTF-8 character into the `utf8_wrapper` at the specific position.

I added code the the input box function that handles the special keys "Home", "End" and "DEL".

----

SDL2 behaves a bit different. Unicode input is reported via the `SDL_TEXTINPUT` event. The input string is stored in a byte array, encoded as UTF-8. That array needs to be split because a single call to `getch` should only return a single byte from this array. And `getch` is the common interface that the SDL port has to simulate.

Example: input is a single Unicode character, encoded as UTF-8 in 3 bytes. The current code would only forward the first byte of that array. The other bytes are completely lost. This implementation stores the whole UTF-8 string, and removes and returns the first byte on each call to `input_manager::get_input_event`. This makes it compatible with curses.

Bonus: function keys can be used with SDL, as I added the translation of those SDLK_* constants to curses constants.

----

The UTF-8 input applies to the input string box (e.g. when carving something), and to the uimenu filter string, to the new world name, new character name. The last two places have a restricted functionality as some keys (like cursor) are used for other purposes in those screens.

Also this enables *all* characters for the player name (`is_char_allowed` is not called anymore as it discards most Unicode characters).

Unicode is not allowed in world names as it is used for file names. But this is actually very backwards. Most current file systems should handle Unicode file names just fine, so I opt to remove that restriction (but keep the specific character restrictions for slash/backslash).