pfsense/pfsense — pull request #1276 by taguchi-ch (2014-09-11 15:41:48+00:00)
Title: Latest Japanese language files and fix multibyte character issues

I checked and modified the machine translated Japanese language file. So submit latest version.
In the process of testing language file on the pfSense box, some issues occurred due to the multibyte character code. One of the issues, multibyte characters cannot be shown correctly in some cases. And another big issue, using multibyte characters sometimes cause to break config.xml.
Therefore, I modify some codes the following:

(1) Modify to use xhtmlspecialchars, xhtmlspecialchars_decode, xhtmlentities and xhtml_entity_decode instead of htmlspecialchars, htmlspecialchars_decode, htmlentities, html_entity_decode.
Htmlspecialchars, htmlentities and html_entity_decode use ISO-8859-1 for default encoding on PHP5.3.x. And this cause to break config.xml when switch language using different encoding.
Now Pfsense has upgraded 5.3.x to 5.5.x, and this problem might seems not so serious. But this type wrapper is very useful for the system management written by PHP. Because PHP specifications change very often, so using default codeset become some risks of the system. That is why many i18n PHP systems take this type wrap scripts.
For security reason, htmlspecialchars, htmlspecialchars_decode, htmlentities and html_entity_decode need to set correct second argument $flags. Therefore I add new functions xhtml* to set default $flags to ENT_QUOTES | ENT_XHTML and $encoding to system_get_language_codeset in pfsense-utils.inc.

(2) Add/Modify some language and character-code relative codes.
e.g.
In usr/local/www/guiconfig.inc and usr/local/www/system.php
-set_language($g['language']);
+set_language($g['language'], system_get_language_codeset());

In etc/inc/authgui.inc
-<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
+<html xmlns="http://www.w3.org/1999/xhtml" lang="<?=system_get_language_code();?>
xml:lang="<?=system_get_language_code();?>">

In etc/inc/pfsense-utils.inc
add some codes to set default language in function system_get_language_code()
+$config['system']['language'] = "en_US"; // Set default language.
+set_language("en_US", "UTF-8");

(3) Unify character-code to UTF-8.
Convert usr/local/share/mobile-broadband-provider-info/iso_3166-1_list_en.xml from ISO-8859-1 to UTF-8.
Set pt_BR encoding from ISO-8859-1 to UTF-8 in etc/inc/pfsense-utils.inc and pfSense.po of pt_BR.
Copy pt_BR language files usr/local/share/locale/pt_BR.ISO8859-1/LC_MESSAGES/ to usr/local/share/locale/pt_BR/LC_MESSAGES/.
Convert usr/local/share/locale/pt_BR/LC_MESSAGES/pfSense.po from Mac OS Roman to UTF-8.
Regenerate usr/local/share/locale/pt_BR/LC_MESSAGES/pfSense.mo using converted pfSense.po.

(4) Modify to use UTF-8 writing config.xml
In etc/inc/xmlparse.inc and etc/inc/xmlparse_attr.inc
e.g
-$xmlconfig .= "<$ent>" . htmlentities($val) . "\n";
+$xmlconfig .= "<$ent>" . xhtmlentities($val, ENT_QUOTES | ENT_XHTML, 'UTF-8') . "\n";

(5) Add Japanese to languege list in etc/inc/pfsense-utils.inc

(6) Fix broken html form layout when switch language
In usr/local/www/services_dhcp.php
-00:00:00,01:E5:FF
+00:00:00,01:E5:FF