From: anoosh@mips.com (Anoosh Hosseini)
Newsgroups: soc.culture.lebanon
Subject: Re: Some thoughts on Arabic aphabet news reader
Keywords: ARABIC PERSIAN GUI COMPUTER
Date: 6 Oct 92 17:33:03 GMT
Organization: Silicon Graphics Inc, MIPS Technologies Inc.
NNTP-Posting-Host: wildcat.mti.sgi.com


Bassem writes:

>First, I recall this topic came up before, cross-posted on SCI.  It may
>be worthwhile to investigate where the previous effort has lead before
>we spend energy on a new effort.

Sorry for delay, read below. Summary: it is in the works.

>Having said that here are the issues that must be addressed:
>
>- Will the Arabic/Persian capability be an extension of an existing news
>  reader, a completely new reader, or a stand alone (off-line so to speak) 
>  application that takes a messages file as input and produces a bunch of 
>  messages ready to be posted.

Each environment has is pluses and minuses.  There are very good news
readers for X11, with source, this is not true for the PC. Etc....

>  If we extend an existing news reader, we are thus limited to systems where
>  news readers are already developed (UNIX, vms, etc..)  Even then we are
>  only interested in a subset of these capable of bit mapped graphics display.
>  This effort benefits very few people and is thus limited in scope and is 
>  perhaps not worthwhile in the short run.  In the long run, it could make
>  use of a "Language: " field in the header to trigger it.
>
The language field will be part of the header, (actually encoding rather
than language).  It will be up to the local system to attempt to 
interpret the data.
 
>  To build a completely new reader is even more extensive and will 
>  take even more effort.

True. Don't reinvent the wheel.

>
>  The best short term solution is thus to build a stand alone application that
>  takes an input file which contains several messages saved from a news reader
>  in email format and let a user process those messages and save replies in
>  individual messages.  A generic program may be written with extensions for
>  Mac, PC, X, so on.  You compile whatever extension works for you.  You then
>  have to read your news as usual, save Arabic messages for further processing,
>  start the Arabic reader, load the Arabic messages file, reply to each message
>  and save each reply in a separate file, run your news reader, load and
>  post each message individually and voila.  Cumbersome I agree, but it will
>  work for most people.
>
Our initial approach is on similar ground, there are tradeoffs all around.
Note that we decided for the PC not because of the news reader only, but a goal
of developing many other applications. Thus the new reader is only a 
vehicle for us to develop the technology. (with the source in our own 
hands).  At the time of paper design, PC/MAC was the only
common denominator.  Also initially we started parallel efforts in X11 and on
the PC, but due to lack of manpower, we focused on the PC. 

>  On some systems, it may be possible to change the display fonts on the fly
>  and thus switch between the Arabic/English soft fonts with few keystrokes.
>  The problem with such a scheme is that you will really have to learn to
>  read and write backwards.
>

We have the routines for graphics screen management with variable size fonts.
(I don't like Farsi/Arabic fonts which are squeezed into the n x m Latin
matrixes. The VGA card was optimized for the display of Latin, not Arabic.)
However this is the path taken by my who want to "localize" latin software.
(It makes life much easier.)

>- Next issue is how you encode the Arabic alphabet in ASCII.  The options are
>  as follows:
>  . Use some coding set which bears no resemblance to Latin characters
>  . Create a one-to-one mapping between Arabic and Latin (with extensions) that
>    is somewhat readable (for example the Latin "L" is the Arabic letter "lam"
>    and with upper case or exotic characters completing the Arabic letters not 
>    found in Latin).  Double characters here (for example the "kh" to stand
>    for Arabic "kha" as in khalil) is not allowed for the obvious reason (what
>    do you do the the sequence kaf-heh?)
>  . Allow doubling one-to-two mapping with an escape character to signal a
>    double character.
>
We had our own escape sequence method, but with new specs coming out, I
think we will go with Internet standards.

>- Next issue is whether you include alternate forms of a message on the same
>  post.  This issue is related to the previous issue.  If we use an arbitrary
>  coding set which bears no resemblance to Latin then in fairness to those who
>  are stuck with character-mapped terminals, we need to include two forms of 
>  the message in the same post: A Latin-readable form and the coded message.
>  Some keyword then marks the end of the first form (the Latin-readable form)
>  and the start of the coded form.

The English part will always be readable, and required for transport on
Internet.  Use of Persian/Arabic with be still in Latin characters, but 
that's for transmission only, they would be representing ISO 8859-6 
encoding or equivalent. Read below.

>
>  A more reasonable approach is to stick with the less readable one-to-one
>  mapping and forgo with the two forms of a message in the same post.

Ideally you would want to send 8 bit data, but Internet only guarantees 7 bit
transportation at this time. For local systems which support 8 bit 
data, this is not an issue.
 
>
>  The one-to-two with an escape character is also possible.  It would be more
>  readable but somewhat more cumbersome to use by someone without an Arabic
>  reader.
>
>
>Of all the option I posted above, the easiest to implement is thus a stand
>alone display/reply application (which incidentally doubles as an Arabic 
>sort-of-email application) with a one-to-one somewhat readable Latinized
>coding set, or a one-to-two with an escape (quoting) character.
>
>bassem
>
>
	we have a demo model, but its not complete, the Persianization 
needs a little cleanup. My colleague Bahman with a team has developed a
custom news reader on the PC which works on a file structure similar
to the one on UNIX.  If one has PC-NFS it can work directly from the
UNIX tree. I  hope to have my responsibilities done by early 93. 

	Things have changed since we first started.  There are several
approaches, they share many aspects, however each solution has different
tradeoffs. The first conclusion of our work was that you cannot make everyone
happy, but you still go ahead, and try to develop the best engineered
solution. We firmly believe in visualizing Persian/Arabic, the use of
variable size fonts and with the best display presentation/cost ratio.   
At the time, 2 paths made sense, one supporting X11 and the other MS windows.  
I was most familiar with the PC world, and that's how I decided to go for 
the PC rather than the Mac.
Since then Apple has released Arabic and Persian extensions to their OS, and a
friend of mine is using their toolset to develop Persian software. I
will let him discuss this own work.  Microsoft is working on Arabic
Windows, (to be released some time this year). With the aid of their
tools it will be easier to develop such applications.  In our project
we started from scratch and developed all the screen management routines,
and we hope to use this as an edge.

In the same time period, Sun continued to dominate the Workstation market, 
but now they support X11. So X11 will continue to be the path for future work.
8 bit clean UNIX systems are now standard, before you had to get
custom versions of UNIX.  Any implementation of SVR4 has national
language support and a methodology for Internationalization.  There
are Internet specs now discussing Internationalized mail, (extensions
to RFC 822, which only allows 7 bit printable ASCII characters.) Also
methods for transporting 8 bit/16/32 bit encodings on systems which 
still have 7 bit transport mechanisms. 

Arabic/Persian News reader approaches:

The first step is to agree on encoding of messages, independent of how
we display the messages.  For Arabic, ISO 8859-6 is the logical choice.
Persian also has a standard, however it is still not internationally
recognized.  ISO 8859-6 requires the full spectrum of the 8 bit encoding.
Thus UNIX SVR4 machines will have no trouble handling 8859-6 data. 
The problem lies with News Readers and the rest of the Internet.  Current
news readers filter all non-printable characters out and some sites 
only allow the transportation of 7 bit data. (that's why we have uuencode
etc...)  Recently these matters have been addressed, refer to: 

RFC1341: MIME  (Multipurpose Internet Mail Extensions):
Mechanisms for Specifying and Describing the Format of Internet 
Message Bodies.

RFC1342: Representation of Non-ASCII Text in Internet Message Headers

These RFC's are not final yet.  We were planning our own scheme for
the encoding of 8 bit data in the posting / email, but will support
Internet standards which have equivalent functionality. Note that these
specs discuss our problem in terms of ISO 10646 / Unicode etc....
 
Now assume we are in agreement on encoding, lets move to display of 
Arabic. 

On UNIX workstations running X11:
Rewrite the Athena/Motif widgets to be bilingual, then by linking with
the existing XRn New reader , we have a very productive tool. This 
methodology can apply for xmh (mailer) and other public domain packages.
IBM/HP are both developing Arabic extensions to X11, but this is mainly
to increase market share, the source code with not be released to other
vendors.

On 8 Bit clean UNIX systems:
By having a bilingual Xterm/vt100 emulator, we can have a very 
functional bilingual Unix. This would mean vi, mail, emacs and rn would
all function correctly.  Either Xterm can be redone or have a software
emulator on the PC/MAC. There is now vt100 emulator sources for the
Windows  (MAC ?). 


ON PC/MAC:
Have a custom/public domain news reader which either has source code,
or the user can specify his/her choice of  "editor"  for viewing 
messages. Replace editor with custom bilingual editor. Files are brought
onto the PC/MAC through several channels. It is even possible to do
NNTP over serial lines. Lots of options here.


I have just touched the surface of the topics, let me know which
areas you want to get into deeper. As for our project, if it was a job,
I would have been fired by now. :-) With aggressive schedules at our
regular employment, we try to place as much spare time on the project as we
can spare.  So progress is slow.  For a final political statement,
(got to have one if you post to soc.*) If we just had one less tank in
the Middle East, it could fund several people to work on this stuff
full time and make a difference.  It is a shame/embarrassment for me
to answer questions regarding Persianization/Arabic to foreigners
(Japanese for example), though we are all professionals, finding out 
that they have funding and a team of graduate students working on 
the project, while we have to start 10pm and hit the bed 1:30am so 
that we can go to work the next day. 


best wishes,

-anoosh


-- 
-----------------------------------------------------------------------
| Silicon Graphics Inc./MIPS Technology Inc.   	| anoosh@mti.sgi.com  | 
| 2011 N. Shoreline Blvd. M/S 10U-178           | anoosh@sgi.com      |
| Mountain View, CA  94039 USA.               	| 415-390-4497	      |
Newsgroups: soc.culture.lebanon