xp-framework/rfc — Issue #146: Unicode
Author: thekid
Created: 2011-10-17 16:00:36+00:00
State: open
Labels: draft, framework

Scope of Change
====
The XP framework will fully support Unicode.

Rationale
====
Internationalization.

Functionality
====

General overview
----

```
* We will use utf-8 as internal encoding
* All strings will contain utf-8
* Thus, all source files will need to be in utf-8
* Bytes != Characters: The capital german umlaut "U" (Ü) is one character
but represented by two bytes in utf-8!

```

Platform encoding
----
The platform encoding is determined by querying the environment with:

```php

```

Values:

```
Platform env.LC_CTYPE env.LANG loc.LC_CTYPE
====
WinXP Cygwin terminal (n/a) C German_Germany.1252
WinXP Cygwin uxterm (n/a) en_US.UTF-8 German_Germany.1252
WinXP cmd.exe (n/a) C German_Germany.1252

FreeBSD 6 via ssh (n/a) C (n/a)
FreeBSD 5 via ssh (n/a) C (n/a)

Debian 3.1 via ssh (n/a) C C
Debian 4.0 via ssh (n/a) C C

```

Strings
----
The framework already contains supporting classes to represent strings
and characters, both of these can handle unicode characters transparently
(lang.types.String, lang.types.Character).

```php
length(); // 9
$s->charAt(0); // lang.types.Character<"Ü">
$s->charAt(0)->getBytes(); // 2 bytes (\xC3 \x9C)
$s->getBytes(); // 10 bytes (\xC3 \x9C b e r c o d e r)

$s= new String('Ãbercoder'); // same as: new String('Ãbercoder', 'utf-8');
$s->length(); // 9
$s->charAt(0); // lang.types.Character<"Ü">
$s->charAt(0)->getBytes(); // 2 bytes (\xC3 \x9C)
$s->getBytes(); // 10 bytes (\xC3 \x9C b e r c o d e r)
?>
```

Primitive strings *must* be written in utf-8:

```php

```

Binary data
----
Binary data (e.g., the contents of a JPEG image) may not be en- or decoded
in any way. Thus, we will add a new class to represent binary data.

TODO: Name? Suggestions:
* lang.types.ByteArray
* lang.types.Bytes
* lang.types.ByteList
* lang.types.Binary

Regular expressions
----
There will be supporting classes for regular expressions modeled after
the idea in http://experiments.xp-framework.net/?arena,regex

For all preg_* functions, the /u modifier exists for unicode.

XML
----
The default encoding of all XML documents will be changed to utf-8.

Databases
----
The database APIs will be changed to set the client character set to
utf-8. By doing this, the server will automatically convert data from
and to this character set if necessary, thus making no further client-
side conversions necessary.

* MySQL:
After connecting, the SQL query 'set names "utf8";' will be sent.

* Sybase:
The character set parameter of sybase_connect() will be set to the
string "utf-8"

* PostgreSQL:
After connecting, the SQL query 'set names "utf8";' will be sent.

* SQLite:
The encoding can only be set once for the main database by using
'PRAGMA encoding = "UTF-8";'. Once it's set, it can not be changed.

* MS-SQL:
TODO

```
MySQL versions < 4.1 do not support 'set names "utf8";' and Sybase
must have character set conversions sp_configure'd - what should
we do if setting the client character set fails? Fail to connect? Add
a client-side conversion?

```

LDAP
----
LDAP already stores everything in utf-8, so the de/encoding process can
be removed.

Scriptlets
----
Scriptlet's content-type headers will be set to

```
Content-Type: text/html; charset="utf-8"

```
per default. XMLScriptlets already honour the charset given in

```

```

Files
----
Files without explicit encoding (text files) will be assumed to be in
platform encoding.

File names
----
File names will be assumed to be in platform encoding.

Note:
Windows supports Unicode in filenames, see

http://msdn2.microsoft.com/en-us/library/aa365247.aspx
"Naming a File"

PHP cannot cope with this, see
http://bugs.php.net/30195
"scandir etc cannot read Chinese file/folder name"

Because PHP's directory and filesystem functions report unicode characters
in filenames as question marks, file names like this:
[[unicode-filenames.png]]
cannot be supported.

The following Java sourcecode:

```php

```
...yields the same problem. If the VM is started with "-Dfile.encoding=utf8"
though, the filenames are utf8-encoded *twice* (and will display correctly
in a Unicode-aware terminal but *not* in others).

Python can handle unicode filenames *if* the argument passed to os.listdir()
is of type unicode:

```php

```
Note this will not work using Cygwin-Python.

Console
----
* Input parameters are encoded from platform encoding to internal encoding.
* Output is encoded from internal encoding to platform encoding.

The following script: (echo-param.php)

```php
value(1));
?>
```

When run inside a Windows shell on German Windows setup the XP framework
will:
* Detect $ctype= "German_Germany.1252" via setlocale(LC_CTYPE, 0)
* Set input and output encoding to "CP1252"
* Convert values in $argv from CP1252 to utf-8 (= internal encoding)
* When writing to the console, convert from utf-8 to CP1252.

Everything will work as expected:

```
C:\> php echo-param.php "Übercoder"
Param: Übercoder

```

When run inside a cygwin "uxterm" shell on the same machine the XP
framework will:
* Detect $ctype= "en_US.UTF-8" via getenv("LANG")
* Set input and output encoding to "UTF-8"
* Not convert anything.

```
$ php echo-param.php "Übercoder"
Param: Übercoder

```

Security considerations
====
n/a

Speed impact
====
Slightly slower when converting data.

Dependencies
====
* iconv extension (default since PHP5)

* This will bump the major version number (the first unicode release
would be 6.0.0)

Related documents
====
- http://xp-framework.net/rfc/contrib/rfc0146.diff
Implementing patch

- http://de3.php.net/iconv
PHP's iconv extension

- http://java.sun.com/docs/books/tutorial/i18n/index.html
Java Tutorials: Internationalization

- http://www.microsoft.com/globaldev/getwr/dotneti18n.mspx
Microsoft .NET Internationalization

- http://www.python.org/dev/peps/pep-0100/
PEP 100: Python Unicode Integration

- http://www.amk.ca/python/howto/unicode
Unicode HOWTO (Python-specific, also discusses unicode in general)

- http://www.rooftopsolutions.nl/blog/filesystem-encoding-and-php
Filesystem encoding and PHP