evanmiller/ChicagoBoss — pull request #89 by ztmr (2012-03-14 12:04:52+00:00)
Title: Unicode support

Hello,

I was playing with ChicagoBoss Example Wiki as I am working on project which will work on the same principle as Wiki, but from a bit different point of view.

As I wrote some regular expressions to implement my markups, I found out that my regexp does not work properly with diacritics-contained strings -- even if I turned on re:'s Unicode support. In one moment, I thought Erlang's Unicode implementation is broken (it is relatively young).

Finally, I have found, that strings stored within my database (Mnesia) are not Unicode! So I did some tests and figured out that the entire problem is in Misultin request processing, not in Erlang itself.
Unless you boss_db:find() your record by hand from Erlang shell, diacritics is working OK. But if you try to write some data from database on UTF-8 compatible terminal, it will show trash.

I have fixed this on the side of request and it is a subject of my pull request.

Now we will have to think about solution for response, because it is very bothering to use unicode:characters_to_binary() still again and again for each string I need to return to client.
As temporary solution, I made some functions into my model, for example for Text field, I made utext()
to return unicode:characters_to_binary (Text). This allows me to use {{ page.utext }} instead of {{ page.text }} (which will crash!) in my ErlyDTL views.
But this is only temporary and not acceptable for future use of Unicode in conjunction with CB.

How to get rid of these complications? Should we implement some Unicode decision logics in ErlyDTL's variable expansion? I think it should be much more clear to implement it in Misultin Response. What would you recommend?

Thanks,
Tom.

PS: Maybe, we should wait with my pull request unless we figure out (and implement, of course :-) ) some Unicode support for Response too.