HDTenEightyP/NetNews huggingface dataset:
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Language tagging
Date: 4 Jan 93 18:08:10 GMT
Organization: Tokyo Institute of Technology

In article <1336@blue.cis.pitt.edu>
	djbpitt+@pitt.edu (David J Birnbaum) writes:

>It is true that Vadim's character set always carries language
>information along with it, while bare Unicode does not. A Unicode based
>system, though, will require more than the bare character set (which is
>why I refer to it as a "Unicode-based system," rather than simply as
>"Unicode"). 

Until we can figure how to attach the information, we can't use
Unicode or "Unicode-based system".

>Concerning the former, I would normally require that my texts include
>language identification,

So, how can you do that?

					Masataka Ohta
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: An alternative I18N paradigm
Date: 4 Jan 93 15:23:05 GMT
Organization: Tokyo Institute of Technology

In article <24495@alice.att.com> andrew@alice.att.com (Andrew Hume) writes:

>	no, you cannot simply wish the problem away by insisting
>on a single locale for everyone. there simply exist different
>local conventions and formats which may be politically, culturally, or
>legally required. you (we) have to deal with them. the challenge is to
>do so in a comprehensible fashion.

True, but, there shouldn't be any locale in universal character code set.

						Masataka Ohta
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: Unicode ISO10646 CharacterEncoding
Date: 10 Jan 93 03:29:10 GMT
Organization: Tokyo Institute of Technology

In article <1in2c8INNmbj@life.ai.mit.edu>
	glenn@wheat-chex.ai.mit.edu (Glenn A. Adams) writes:

>One should not in general use an interchange code (UTF1 or UTF2) for
>processing.  While one may use a process code for interchange,

That is opposite.

One can use interchange code for processing if it is convenient.

There is no problem in doing so, as each application knows every detail
of how interchange code is.

On the other hand, one can't use process code for interchange, unless
you are living in the closed world, because other applications won't
accept it.

>(e.g., Unicode and 10646 UCS[24] allow NULL bytes and ISO2022 C0/C1 control
>code bytes in any byte position of their "process codes").

That is one of famous fatal design flaw of it.

>I can't imagine why anyone in their right mind would want to use UTF[12]
>or any other ostensibe interchange code for processing, given the problems
>of variable length encodings.

Maybe, you can't. But, in many cases, variable length is of no problem.

>But if one is to create
>an aware application which uses more than the ASCII subset, or if it is
>to memory map files, then use of a fixed-with process code (even for backing
>store) becomes much more sensible.

You should be crazy. If you are mapping a file under possibly networked
environment (these days, all environments are so), you can't use multiple
octet fixed width code because of endeaness. Don't say signature, because
it makes everything complex and, thus, slow, and your reasoning to map
files should be for efficiency and for simplicity. BTW, though file mapping
is neither efficient nor simple, it is another topic.

						Masataka Ohta
From: unrza3@cd4680fs.rrze.uni-erlangen.de (Markus Kuhn)
Newsgroups: comp.std.internat
Subject: Re: Let's develop ISO sorting rules
Date: 7 Jan 1993 15:21:04 +0100
Organization: Regionales Rechenzentrum Erlangen
Reply-To: mskuhn@immd4.informatik.uni-erlangen.de
NNTP-Posting-Host: cd4680fs.rrze.uni-erlangen.de

maf@dtek.chalmers.se (Martin Forssen) writes:

> I'm afraid a lot of Swedes will refuse to accept any sorting order which
> places aring and adiaeresis adjacent to a. In swedish these are completely
> different characters and aring and adiaeresis comes after z. All of this
> IMHO of course.

Isn't the Swedish convention to sort a-ring after z just as arbitrarily
as the confusing old convention to sort a-diaresis in Germany as ae?

There are at least a few real world applications (I don't talk about
trivialities like UNIX ls), e.g. catalogs of international libraries,
where an _easy_ to understand language independend sorting standard would
be extremely useful. Why not standardize one for Unicode in an easy
_implementable_ way? Then let's see, how many people will adopt it and
will quickly forget their traditional rules. Perhaps even many users
in Sweden. There are perhaps already some multilingual sorting rules
common practice, but I know no one that defines an easy implementable
deterministic and total ordering for ISO 10646 strings.

I don't suggest to remove all locale mechanisms, but just to offer one
simple, efficient, easy to understand modern 'international locale' for people
who don't stick to their national locales and who have serious problems
with multiple different locales.

Markus

-- 
Markus Kuhn, Computer Science student -=-=- University of Erlangen, Germany
Internet: mskuhn@immd4.informatik.uni-erlangen.de  |  X.500 entry available
--- Wer, wie, was? Wieso, weshalb, warum? Wer nichts fragt bleibt dumm. ---
From: erik@poel.juice.or.jp (Erik M. van der Poel)
Newsgroups: comp.std.internat
Subject: Re: islands
Date: 11 Jan 93 02:08:00 GMT
Organization: sometimes

> So, what's my plan?  Talk utf and 9P and get some connectivity happening.

Yes!  I like this plan.  Plan 9's UTF (aka FSS-UTF aka UTF-2) is
designed very well, and is clearly a very good solution for such
things as file names and file contents in UNIX-like and Plan-9-like
systems.  Some of the remaining issues to be considered:

(1) What to do about email and netnews?

    I guess you need to choose from the following:

        (a) try to get links to be "8-bit clean"
        (b) don't bother with (a) and encode in 7 bits
        (c) do both (a) and (b), somehow "knowing" which links are 8-bit

(2) What to do about systems that use straight Unicode

    If such systems actually become popular, perhaps design flexible
    interfaces e.g. optional conversion between UTF and Unicode in NFS,
    FTP, etc.

(3) What to do about systems using other encodings, e.g. ISO-2022-JP

    The American infrastructure is large enough that it can safely ignore
    Japan.  If some people within that infrastructure absolutely need to
    read e.g. fj.* newsgroups, they can build ISO-2022-JP <-> UTF
    converters.  If UTF catches on even in Japan, such converters may
    become unnecessary.

(4) What to do about "data announcement" (or "file typing")

    In some protocols, e.g. MIME, you have no choice -- you must indicate
    that the text is UTF.  In other protocols, such as FTP, there is no
    data announcement, so you have to decide whether or not you're going
    to add such an option to FTP.  Roughly the same can be said about NFS.

(5) What to do about language tags in Unicode/UTF

    Probably need to continue researching this area.


The above are in my own prioritization order, but I suspect that many
of you will attach different priorities, or substract from/add to this
list.  Or substitute 9P for NFS, or add 9P to the list, etc.


Respectfully submitted by:
-- 
Erik M. van der Poel                               erik@poel.juice.or.jp
From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages
Date: 7 Jan 93 19:36:07 GMT
Organization: MIT Artificial Intelligence Laboratory
NNTP-Posting-Host: wheat-chex.ai.mit.edu

In article <1993Jan7.065611.15193@fcom.cc.utah.edu> terry@cs.weber.edu (A Wizard of Earth C) writes:
>This example becomes more of a problem when translated to one of a glyph
>variant between Chinese and Japanese.  I agree that the problem is one
>of words, not characters -- however, in ideographic languages, words *are*
>characters.  The example is not as artificial as you make out.

I'm afraid this is incorrect.  Ideographic characters in Chinese, Japanese,
Korean, and Vietnamese (chu+" no>m va! chu+" ha'n) *are not* equivalent
to words.  Rather, they are mostly equivalent to morphemes (although there are
a number of multiple character morphemes).  Many words in these languages
are composed from multiple morphemes, e.g., modern Chinese has about 2.2
morphemes (characters) per word on average.  For example, in Chinese
"zhong1guo2" 'China' and "ri4ben3" 'Japan' are single words but contain
two morphemes each.

Glenn Adams
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: An alternative I18N paradigm
Date: 7 Jan 93 08:31:22 GMT
Organization: Tokyo Institute of Technology

In article <1993Jan7.054124.14059@fcom.cc.utah.edu>
	terry@cs.weber.edu (A Wizard of Earth C) writes:

>>Other kinds of tags might be necessary for certain types
>>of processing, e.g., yomi (phonetic reading) tags for allowing the
>>display of furugana, sort keys for allowing producer specified sorting
>>behavior, and so forth.
>
>Yes!

Wrongo!

Phonetic reading tags is overkill, which make everything too much
complicated.

Making something complex only makes the problem unsolvable.

>>10646 will not even address any of these matters.  However, Unicode may
>>do so in the form of implementation guidelines or further work on I18N.
>
>Yes!

What?

The original DIS 10646 was free from CJK unification defect.

Then, Unicoders made 10646 unusable ignoring many rational oppositions.

>>Nonetheless, I think that it is quite important for many parties to begin
>>implementing such systems so that development of standard tags and tagging
>>systems can proceed.  We need prior art and experience in these areas
>>before effective standards can be developed with a reasonable hope of
>>success.

We have many prior arts in Japan, based on which the original DIS 10646
was constructed.

Unicode, as it is now, is unusable and can't be implemented on a usable
internationalized product.

						Masataka Ohta
From: jenkinsj@blowfish.taligent.com (John H. Jenkins)
Subject: Re: Unicode Han Characters [was Re: Language tagging]
Followup-To: comp.std.internat
Organization: Taligent, Inc.
Distribution: uk, na
Date: Thu, 7 Jan 1993 20:39:43 GMT

In article <1ih0ptINNad8@frigate.doc.ic.ac.uk>, rap@news (Ross Paterson)
wrote:
> 
> From article <1if6lgINN6ri@life.ai.mit.edu>, by
glenn@wheat-chex.ai.mit.edu (Glenn A. Adams):
> | [...]
> | If you would like to obtain (unofficial) mappings between a number
of
> | these character sets and Unicode, you can find them by anonymous FTP
> | on METIS.COM [140.186.33.40] in /pub/csets.
> 
> (Actually /pub/csmaps)
> 
> Does anyone have a mapping from the old Chinese telegraph code to
Unicode?
> 
> 

We do.  I'll check to see if we can make it available.  You should be
aware, however, that it has not been checked as thoroughly as the
mappings between Unicode and national character set standards such as
JIS, GB, and KSC.  


----
John H. Jenkins
John_Jenkins@taligent.com

PS, I couldn't mail back to you.  Please forgive my posting this on the
net.
Newsgroups: comp.std.internat
From: flaps@dgp.toronto.edu (Alan J Rosenthal)
Subject: Re: Let's develop ISO sorting rules
Date: 6 Jan 93 03:26:27 GMT

unrza3@cd4680fs.rrze.uni-erlangen.de (Markus Kuhn) writes:
>Believe me, German users who know what they are talking about won't
>complan if "a, "o and "u are not sorted as ae, oe and ue.

I believe you.  I'm sure that the analogous statement is true for French.  But
I'm almost as sure that Swedish readers won't want "a-circle" to be anywhere
near "a".  I think your flexibility on this is a language-relative phenomenon.

>Sorting all latin characters directly behind their version without a diacritic
>will be accepted by 99.9% of people using latin alphabets,

Let's hear from more Scandinavians on this.. and where would you put the Danish
o-with-a-slash-through-it?  Surely that slash isn't a diacritical mark...

I think that your statement from a later article,

>Word lists produced by my algorithm are pretty easy to scan for human eyes.

is language-relative as well.  If people are used to thinking of a-circle as
being at the end of the alphabet, it may become a different letter.  For all I
know they think of it as a circle with an "a" as a diacritical mark rather than
the other way around.

ajr

--

"quote"
From: unrza3@cd4680fs.rrze.uni-erlangen.de (Markus Kuhn)
Newsgroups: comp.std.internat
Subject: 1 Inch (was: ISO paper sizes)
Date: 5 Jan 1993 11:26:52 +0100
Organization: Regionales Rechenzentrum Erlangen
Reply-To: mskuhn@immd4.informatik.uni-erlangen.de
NNTP-Posting-Host: cd4680fs.rrze.uni-erlangen.de

ag129@cus.cam.ac.uk (Alasdair Grant) writes:

>In article <1i9j4gEINN8nd@uni-erlangen.de> mskuhn@immd4.informatik.uni-erlangen.de writes:
>>1 inch is 25.4000000 mm.

>1 SI inch is 2.54mm _exactly_, not just to 7 decimal places.
                  ^^ you mean centimeters (cm), don't you?

Of course, you are right. My intention was to write

    1 inch is 25.4000000... mm.     :-)

This inch is still used in metric europe for e.g. diameters of water
pipes and most dimensions in electronics are 2.54 mm (1/10 SI inch)
multiplied by something.

>I believe NIST inches are a standard unit in HyTime.

In HyTime (according to a recent CACM article) 1 inch is _exactly_
25.4 mm = 2.54 cm = 0.0254 m. Remember, HyTime is an ISO standard
and ISO standards only use the SI system defined in ISO 31.
(e.g. ISO 31-1:1978 Quantities and units of space and time)

Markus

-- 
Markus Kuhn, Computer Science student -=-=- University of Erlangen, Germany
Internet: mskuhn@immd4.informatik.uni-erlangen.de  |  X.500 entry available
----- Anyone participating in the use of MS-DOS, Heroin or Cocaine is -----
---- simply not getting the most out of life possible. (Brian Downing) ----
From: keld@login.dkuug.dk (Keld J|rn Simonsen)
Newsgroups: comp.std.internat
Subject: Re: Future extension of ISO 10646
Date: 5 Jan 93 00:51:38 GMT
Organization: DKnet

jhelling@cs.ruu.nl (Jeroen Hellingman) writes:

>[about having an A-umlaut for each language seperately to help sorting...]

I believe the sorting procedure is related to the user, not to the data.
It is the user which needs data sorted in the way the user
expect it, so s/he can find what s/he looks for.
For that purpose the character d (a-umlaut, a-diaresis, swedish/finnish
ae) should not have different encodings, according to language, as 
the user does not in many cases know what language the string is
in, for instance is Eberg a Danish or Swedish name. It would be
soretd differently, if it should be sorted according to the
original language.

Most national standards for collating (German, Austrian, Canadian,
Swedish, Danish, Norwegian, Finnish come to my mind) does not
provide specifications for collating according to the original
language, but only with respect to the letters. Some standards are
explicit about this, for example the Danish DS 377 standard.

So I would recommend *not* to have different versions of Latin
characters outside the BMP of 10646. 

I know to little of the Han/Kanji/Hanzi character collation to
give an educated opinion here.

Keld Simonsen
From: rap@news (Ross Paterson)
Newsgroups: comp.std.internat
Subject: Re: Language tagging
Date: 6 Jan 1993 13:44:15 GMT
Organization: Dept. of Computing, Imperial College, University of London, UK.
Reply-To: rap@doc.ic.ac.uk
NNTP-Posting-Host: peaberry.doc.ic.ac.uk

From article <MELBY.93Jan6113951@dove.yk.fujitsu.co.jp>, by melby@dove.yk.fujitsu.co.jp (John B. Melby):
| If what I have heard about the Unicode standard is accurate, Chinese
| simplified forms are not distinguished from Chinese unsimplified forms
| when they are effectively equivalent.  Since the radicals for "eating"
| and "metal" have simplified forms radically differing from the unsimplified
| printed variants, a Japanese may not be able to distinguish certain
| characters displayed using a Chinese character set (although usually
| they should be distinguishable through context).

No, they are distinguished, as are the different forms of the "door",
"speech", "silk", "money", "vehicle", "horse", and "bird" radicals and
several more.  About the only radical where the simplified form doesn't
get a different codepoint (unless something else in the character changes)
is the "grass" radical -- "+ +" (4 strokes) vs "++" (3 strokes) -- but
that shouldn't interfere too much with legibility.

In any case, won't it be possible to specify in a MIME header which
language the text is in, or which presentation conventions are preferred?
--
Ross Paterson <rap@doc.ic.ac.uk>
Department of Computing, Imperial College, London SW7
Newsgroups: comp.std.internat
From: jenkinsj@blowfish.taligent.com (John H. Jenkins)
Subject: Re: Unicode and EACC
Followup-To: comp.std.internat
Organization: Taligent, Inc.
Date: Thu, 7 Jan 1993 20:44:10 GMT

In article <1ig1viINN8an@shelley.u.washington.edu>,
sandstro@cac.washington.edu (Bob Sandstrom) wrote:
> 
> I'm trying to pin down some facts regarding character encodings.
> 
> My copy of The Unicode Standard [volume 1] 
> has the number 13481 for Z39.64, and 13053 for Big Five.
> My copy of The Unicode Standard [volume 2]
> has the number 13053 for Z39.64, and 13481 for Big Five.
> The change does not appear in Volume 2, Appendix D.
> Please explain.
> 
> Also, I took some notes from Z39.64 itself, ISSN 1041-5653,
> as published by Transaction Publishers.  [...] 
> This is a total of 13321 on the Chinese layers, and 15686 on all
layers.
> Where does the 13481 quoted by Unicode come from? 
> 

There are two factors here.  One is a typographical error on page 20 of
volume 2 which we didn't catch in time.  Volume 1 and Appendix D of
volume 2 are correct.  

The second is that Unihan is not a superset of Z39.64.  We do not
guarantee that everything in Z39.64 is represented in Unicode by an
independent character.  This is guaranteed only for JIS, KSC, GB, and
portions of CNS.  There may be 15686 characters in Z39.64.  Only 13481
were considered distinct. 


----
John H. Jenkins
John_Jenkins@taligent.com
From: peter@ferranti.com (peter da silva)
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: ISO10646 Unicode
Organization: Xenix Support, FICC
Date: Mon, 4 Jan 1993 17:35:38 GMT

In article <1i2m57INN4vr@rodan.UU.NET> avg@rodan.UU.NET (Vadim Antonov) writes:
> In article <id.68CW.A16@ferranti.com> peter@ferranti.com (peter da silva) writes:
> >In article <1i13rrINNars@rodan.UU.NET> avg@rodan.UU.NET (Vadim Antonov) writes:
> >> We were talking about lexicographical sorting, not abouth phonetics.

> >But lexicographic sorting (actually, lexicograhic ordering) is a minor part of
> >this. Most sorting computers do is algorithmic ordering, to optimise some
> >combination of operations on data structures (searching, for example). The
> >character set is irrelevant there.

> Wrong-o. Nobody does numerical sorts since invention of secondary
> indices.

I'm afraid you'll have to translate this sentence. It parses as valid English,
and uses appropriate sentence structure and terminology for the context, but
seems almost completely irrelevant to anything I said. For efficient lookup
the index needs to be ordered in some fashion whether it's a flat table or a
tree. Unless you find hashing adequate for all possible purposes, perhaps?

> The problem is not in searching -- the problem is in presenting
> the information and in regular expressions ([a-z] - does it include "o?)

No. The regular expression '[a-z]' is a side effect of the fact that ASCII
happens to be in numerical order for the base alphanumeric characters used
in English computer text. It's invalid for EBCDIC, for example. The POSIX
alternative for what you *mean* here is something like '[:lower:]', and I
would hope that for the long term this be extended to specify localization
information, for example '[:lower/english/usa:]' so that it would allow
loan words like clich'e or na"ive, or names like 'da Silva' (with a non
blank space between the 'a' and 'S').

Sure, it's a mouthful, so you'd do this:

	setenv LOWER '[:lower/english/usa:]'

You need to do that for scripts, anyway, since you want your program to
continue to work when it's downloaded from some site in Finland and used
in London or Beirut.
-- 
Peter da Silva                                            `-_-'
Ferranti International Controls Corporation                'U` 
Sugar Land, TX  77487-5012 USA
+1 713 274 5180                            "Zure otsoa besarkatu al duzu gaur?"
Newsgroups: comp.std.internat
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Dumb Unicoders
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages
Date: 4 Jan 93 17:17:57 GMT
Organization: Tokyo Institute of Technology

In article <DwqPwB3w165w@blues.kk.sub.org>
	kosta@blues.kk.sub.org (Kosta Kostis) writes:

>UniCode is limited because everything is squeezed into 16-bit, but
>for latin languages (and the like) it's just fine.

>>    In reality it means that Unicode is not a solution for
>>    typesetting.

>Come on. What did you expect? A universal character set is not the
>"solution", it's just intended to help you develop it.

>I agree that UniCode is not very good for Asian languages, but
>for European languages (and some more) it's really OK.

Then, could you get out of here, comp.std.internat?

>10 bits? You want to keep Asian languages out of the game, too?  :-)

Or, you want to keep Asian languages out of the game, too?  :-|

>Should
>we ever decide to use the full 32-bits ISO 10646 intended to
>allocate for character codes, we should be able to cover almost
>all languages (to some extend).

Full 32 bit? 18 or 20 bits will be suffice for the real universal
character code set.

>Who defines "required"? This is a classical "don't care about the
>users culture/needs" approach that was so harmfull in the past.

ISO 8859-1 is not very good for all European languages, but
for central European languages (and some more) it's really OK.

Then, you wrote:

	I agree that UniCode is not very good for Asian languages, but
	for European languages (and some more) it's really OK.

What was so harmful in the past? What you are repeating?

>The other languages you stated, like Russian, Greek and English
>seem to be served well by UniCode, I think.

But, as for reg-exp pattern, Vadim's suggestion is quite right.

					Masataka Ohta
Newsgroups: comp.std.internat