colobot/colobot-data — pull request #6 by OdyX (2013-10-25 14:52:27+00:00)
Title: "Help" files and "Level scene description" files gettext-based translation infrastructure

Hi all,

What is this all about
====

The core idea of this bunch of changes is to **move from full-file text translations to a purely gettext-based translation mechanism**. The driving idea behind that is that _gettext is the FLOSS standard for translations_, be it for server applications, desktop applications or (as in this case) games, see for example http://www.wesnoth.org/gettext/ which perfectly demonstrates what gettext can allow to do.

How this is implemented
====

I suggest that the reviewers don't take a look at the big one fat diff (arguably enormous; "Showing 2,129 changed files with 142,001 additions and 5,540 deletions.") but rather take a detailed look at each of the 8 consecutive commits:

fc03c57 Add CMake-Po4a-based infrastructure to allow po-based translation of Colobot help- and scene-description (level) files
----

This is where the complete machinery is created. Let's detail a bit how it works:
* [CMakeLists.txt](https://github.com/OdyX/colobot-data/blob/fc03c57f8e94840280f74e49681ed4afede5b58f/levels-i18n/CMakeLists.txt) is the main driver of that big truck: depending on its input variables (LEVEL_CODENAME,SCENEFILE,SCENEDEST,PODIR,HELPDIR and HELPDEST), it will create various make targets that will launch different script with different argument lists to enable the translation. Let's take a look at the scripts:
* [01_create_po4a_file.sh](https://github.com/OdyX/colobot-data/blob/fc03c57f8e94840280f74e49681ed4afede5b58f/levels-i18n/scripts/01_create_po4a_file.sh) will:
* symlink the files-to-be-translated (english versions) and the translation directory (usually "po") into the CMAKE_CURRENT_BINARY_DIRECTORY; and
* output a po4a-compatible configuration file pointing to both the translation files (po/*) and the files-to-be-translated.
* [02_run_po4a.sh](https://github.com/OdyX/colobot-data/blob/fc03c57f8e94840280f74e49681ed4afede5b58f/levels-i18n/scripts/02_run_po4a.sh) simply launches po4a from the correct directory with the correct arguments. Note that the "-k100" in that script is a "translation percentage" threshold; it currently imposes that only fully-translated files take over the legacy ones (more about that below). It also exports the PERLLIB environment variable to allow the use of two specialized Po4a formats (more about that below again).
* [03_inject_translations.sh](https://github.com/OdyX/colobot-data/blob/fc03c57f8e94840280f74e49681ed4afede5b58f/levels-i18n/scripts/03_inject_translations.sh) outputs one scene-description file that has all the translations of the Title, Resume and ScriptName for that scene.
* [04_move_helpfiles_to_paths.sh](https://github.com/OdyX/colobot-data/blob/fc03c57f8e94840280f74e49681ed4afede5b58f/levels-i18n/scripts/04_move_helpfiles_to_paths.sh) prepares the final installation of the translated help files. This is needed because Colobot wants the helpfiles to be in one-char language identifier directories (E/ P/ F/ D/) when Po4a creates two-chars language identifier-based filenames. This is mostly implemented this way as an effort for backwards compatibility (more about that below).
* [05_move_untranslated_helpfiles_to_paths.sh](https://github.com/OdyX/colobot-data/blob/fc03c57f8e94840280f74e49681ed4afede5b58f/levels-i18n/scripts/05_move_untranslated_helpfiles_to_paths.sh) handles the case where Po4a is not available: in that case all legacy helpfiles are still installed (backwards compatibility again) in the correct directories.

7f3929a Add the help/ and levels/ CMake machinery to enable the po-based translations
----
This implements the correct call for levels-i18n/CMakeLists.txt from both help/ and levels/ directories.

c3fb814 Move level scenes to level-specific directories
----
This is a first step towards having one directory per level (with the idea that user-provided levels could be prepared that way and benefit from the same translation infrastructure). The scene descriptions are all moved to being named "scene.txt". The machinery put in place by the above commits makes sure that Colobot still gets to see level named as previously (backwards compatibility again).

04a9e8f Correct the Polish files capitalisation to be uniform across languages
----
This is a bugfix-only commit. As the whole machinery works based on filenames, some Polish files were wrongly capitalized, creating weird-to-debug bugs or missing files all around. This commit can be cherry-picked right away on dev without impact on the course of the program

876a171	Move help files to level-specific directories
----
This is the continuation of c3fb814, but this time for the multilanguage help files. Although their sources are moved in the level-specific directories, they are put back in the global help directory by the machinery. Backwards-compatibility again :-) .

768bb85 Add translation files for all levels
----
This is where all *.pot and *.po files are added to all levels. As you can see, they are mostly empty but thanks to the threshold level (100%), the legacy full-file translation files are used instead. Note that the files are still needed for the machinery to take a look at the original files.

fe6800d Move the generic helpfiles into categories: generic, object, bots and cbot
----
This split is intended as a way to reduce the amount of translatable strings into categories. The machinery again pushes the files back to their original places.

fb5bdf0 Add translation files for all categories
----
Same as 768bb85, but for the help categories.

Backwards-compatiblity and _source vs built_
====
This change arguably brings one big thing: the source data/ directory cannot anymore be used as a datadir for a built colobot. Although this could be seen as a drawback, I think it's a needed step to take. This had already started with https://github.com/colobot/colobot-data/pull/4 and is just a further step. One other thing to consider is that the colobot-data repository must always stay a _source_ directory, in which the "preferred form of modification" for data files are stored. Having a build-process convert these files in the most suitable form for consumption by the colobot executable is most wanted anyway I would say.

Another thing to consider is that the data directory becomes usefully buildable only on platforms with an available Po4a. That's IMHO a minor problem as I've made extra efforts to get a minimally useable (no translations) data directory when Po4a is not available.

Another word about backwards-compatibility: although help files have moved quite a lot (to both categories and level-specific directories), the machinery brings them back to their original places after translation (on installation). The same applies for level scene description files; they have been uniformally renamed to scene.txt but they get to their original places in the built directory. By merging these changes, the output of the *make install* process should be bit-by-bit equivalent to what it was before. The point being that *this mechanism is an enabler for future translations*.

Legacy vs translated files
====
As mentionned, all legacy translated files are installed in the destination directory unless they are *fully* replaced by a gettext-based translation. This allows to keep the existing french, german and polish translations without any extra effort. On the other hand, it allows very easy addition of new languages for people with good gettext fluency.
(As a side note, I think it would be worthwhile to get rid of these legacy translations and push them to the gettext files, but as that would be quite a bunch of work, I favoured the backwards-compatibility.)

Parsing Colobot help files
====
One important part of this worl has gone into (parsing the Colobot help files)[https://github.com/OdyX/colobot-data/blob/fc03c57f8e94840280f74e49681ed4afede5b58f/levels-i18n/scripts/perllib/Locale/Po4a/Colobothelp.pm#L131] when handling the translation files (.po). The main reason behind that is that backslash-based marking doesn't fit well into gettext files, where they have to be escaped. I favoured a pseudo-HTML syntax. Note that there's also a (Colobot level file parser)[https://github.com/OdyX/colobot-data/blob/fc03c57f8e94840280f74e49681ed4afede5b58f/levels-i18n/scripts/perllib/Locale/Po4a/Colobotlevel.pm] but it is much simpler as it only takes out the titles (as the rest is language-agnostic)

How the hell can I review all this?
====
Take your time! :-) And ask! I will definitely insist in basing level and help translations on gettext but I can live with a different approach; just take into account that I have put quite some time and effort into this and would appreciate a serious review rather than an upfront refusal.

Cheers, and thanks for considering,

OdyX