HDTenEightyP/NetNews huggingface dataset:
From: unrza3@cd4680fs.rrze.uni-erlangen.de (Markus Kuhn)
Newsgroups: comp.std.internat
Subject: Let's develop ISO sorting rules
Date: 5 Jan 1993 11:03:24 +0100
Organization: Regionales Rechenzentrum Erlangen
Reply-To: mskuhn@immd4.informatik.uni-erlangen.de
NNTP-Posting-Host: cd4680fs.rrze.uni-erlangen.de
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages

bad@flatlin.ka.sub.org (Christoph Badura) writes:

>>I do not think you understand.  From the AVON (Amtliches Verzeichnis der
>>Ortnetzkennzahlen) edition 1985, which gives area codes for the places in
>>Germany.  The next is a selection of places mentioned ("o is o-umlaut):
>>	Modautal
>>	M"ockm"uhl
>>	...
>>now come up with a coding that allows this (standard German) sorting.

>This is _not_ standard German sorting, it's standard German PTT
>sorting (and quite confusing at times).

Well, this is the "Deutsches Einheits ABC" (translate this with German
Standard Sorting) defined in DIN 5007 which clearly defines that
a-Umlaut is sorted as ae. I don't like this rule very much, but at the
moment, it is the most official standard (DIN is our ISO member body)
we have.

>The Duden treats umlauts as the corresponding vowel with respect to
>sorting order (i.e. "o sorts the same as o etc.)

And many other dictionaries ignore DIN 5007 as well, because sorting
a-Umlaut ("a) as 'a' instead of 'ae' makes searching for humans
much more easier.

All this only supports my conclusion again:

We have to define international multilingual sorting rules for Unicode
strings that depend NOT on the language and that do NOT try to be
compatible with every existing sorting system, because nearly no
country seems to have well established sorting rules. This seems to
be the only practical approach in my eyes. I see no disadvantages.

Believe me, German users who know what they are talking about won't
complan if "a, "o and "u are not sorted as ae, oe and ue. Sorting all
latin characters directly behind their version without a diacritic
will be accepted by 99.9% of people using latin alphabets, because
this rule is easy to learn, easy to understand and language
independend. Then let's insert the cyrillic letters (like in KOI-8?)
and the greek letters (delta sorts near D etc.) in this alphabeth
and we get a beautiful multilingual sorting! Let's start to work on
ISO sorting rules.

THE SOLUTION IS SIMPLE ONCE YOU ACCEPTED THAT INCOMPATIBILITY WITH
EXISTING HISTORICAL RULES IS NECESSARY!

Markus

-- 
Markus Kuhn, Computer Science student -=-=- University of Erlangen, Germany
Internet: mskuhn@immd4.informatik.uni-erlangen.de  |  X.500 entry available
----- Anyone participating in the use of MS-DOS, Heroin or Cocaine is -----
---- simply not getting the most out of life possible. (Brian Downing) ----
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Language tagging
Date: 4 Jan 93 17:25:55 GMT
Organization: Tokyo Institute of Technology

In article <1993Jan3.203017.232@enea.se>
	sommar@enea.se (Erland Sommarskog) writes:

>This I don't understand. If you throw away a short e-mail message
>to a friend, why would it be necessary to tag the text with the 
>language?

Why not?

If I throw away a short e-mail message from Japan to Japanese
friend living in China, how can my friend read the text with
the appropriately shaped character?

What if, the mail is stored in some mail box, mixed with Chinese
mails? How can he print the entire mail box?

					Masataka Ohta
Newsgroups: comp.std.internat
From: walters@HQ.Ileaf.COM (Tim Walters)
Subject: Re: ISO paper sizes
Organization: Interleaf, Inc.
Date: Wed, 6 Jan 93 16:02:17 GMT

In article <1ict41EINN7qk@uni-erlangen.de> mskuhn@immd4.informatik.uni-erlangen.de writes:
}walters@HQ.Ileaf.COM (Tim Walters) writes:
}>Printers in Germany and Japan use different sizes for the B series.
}>The German value is specified by DIN; I believe the Japanese value is
}>specified by some JIS standard:
}
}>        DIN             JIS    
}>B4      250x353         257x364
}>B5      176x250         182x257
}>B6      125x176         128x182
}
}That's strange news!
}
}However, the DIN standard says that it is compatible with the ISO standard,
}only the tolerances allowed by DIN are smaller than those allowed by ISO,
}which isn't an incompatibility.
}
}How have the JIS people defined the B series? In ISO and DIN, it is
}just the geometric mean between two A series sizes (geometric mean of
}a and b is sqrt(a*b)).
}
}Still confused ...

Sorry, I don't have the formal JIS standard, I only know that two
Japanese printers we have inhouse, from IBM and NEC, have B4 paper
trays with the above sizes. The manuals refer to a JIS standard, but I
haven't tracked it down.

}Perhaps you mean the following: DIN/ISO define paper end formats. You write
}'printers ... use'. Do you mean the size of the paper delivered to the printer
}before it is sent to the offset printing machine and cut to e.g. B4? Ok,
}that is not the 'paper end format' that is of interest for the common user.

I mean both the paper tray size and the paper size. I believe that B4
is a popular size in Japan. Could someone from Japan who knows more
about this shed some light on the issue?
-- 
Tim Walters, Interleaf          uunet!leafusa!walters, walters@HQ.Ileaf.com
Newsgroups: comp.std.internat
From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Date: 5 Jan 1993 03:41:46 GMT
Organization: MIT Artificial Intelligence Laboratory
NNTP-Posting-Host: wheat-chex.ai.mit.edu
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages

In article <id.E1FW.PX5@ferranti.com> peter@ferranti.com (peter da silva) writes:
>You have identified two problems with Unicode and ISO 10646: case conversion
>and lexical ordering.

I do not agree.  I believe that Vadim thought that similar glyphs were
unified irrespective of script (e.g., Latin T vs Cyrillic T).  Since
this is incorrect, his claim regarding case conversion is unfounded
(though I must admit their are case conversion problems which I didn't
see mentioned, e.g., Turkish i/DOTTED CAPITAL I -- this is an issue
though a bit different than was claimed).

As for lexical ordering, no character set can solve this problem
unless it is defined for use with a single writing system.  A universal
character set abstracts the differences between writing systems
(i.e., languages) by encoding scripts; thus no universal character
set which encodes scripts can simultaneously define all requisite
lexical orderings (though it might choose one ordering arbitrarily).

Consequently, I would say that these are not problems with Unicode
or 10646; instead, they are problems having to do with text processing
in general.  And, no universal character set which unifies scripts
can solve these problems; furthermore, I would argue that a universal
character set which does not unify scripts (i.e., encodings writing
systems directly) will not only be hopelessly inefficient (because
of the much, much larger encoding space required), but also
hopelessly incomplete (because new writing system are being created
all the time). 

Unicode made the correct choice by encoding scripts independently
of writing system (language/orthography); it also made the correct
choice in determining that the problem of lexical ordering is a
higher-level problem, not to be solved by a character set.

Glenn Adams
From: unrza3@cd4680fs.rrze.uni-erlangen.de (Markus Kuhn)
Newsgroups: comp.std.internat
Subject: Re: Let's develop ISO sorting rules
Date: 5 Jan 1993 20:47:44 +0100
Organization: Regionales Rechenzentrum Erlangen
Reply-To: mskuhn@immd4.informatik.uni-erlangen.de
NNTP-Posting-Host: cd4680fs.rrze.uni-erlangen.de
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages

unrza3@cd4680fs.rrze.uni-erlangen.de (Markus Kuhn) writes:

>My vision is NOT a sorting order that is embedded in the character set.
>That would be too trivial, of course. The Unicode developpers had good
>reasons to embed one into the code table. No, I have a slightly more clever
        ^ 'not' missing here, sorry
>algorithm in mind, that will do 2 passes:

Markus

-- 
Markus Kuhn, Computer Science student -=-=- University of Erlangen, Germany
Internet: mskuhn@immd4.informatik.uni-erlangen.de  |  X.500 entry available
----- Anyone participating in the use of MS-DOS, Heroin or Cocaine is -----
---- simply not getting the most out of life possible. (Brian Downing) ----
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: What are the ISO paper sizes? (Was: An alternative I18N paradigm)
Date: 4 Jan 93 17:05:45 GMT
Organization: Tokyo Institute of Technology

In article <1hvslkEINN2h3@uni-erlangen.de>
	mskuhn@immd4.informatik.uni-erlangen.de writes:

>>|  - use ISO paper sizes (e.g. A4, C5, ...)
>
>>However, what I would like to know is what size is A4 or C5 or the rest
>>of these map references that ISO uses for paper.
>
>As far as I know, these paper sizes are not only used by ISO, but by nearly
>all countries on this world apart from North America and Japan.

While I personally think letter size is sometimes esthetically better
than ISO, ISO paper sizes are dominant also in Japan.

						Masataka Ohta
From: djbpitt+@pitt.edu (David J Birnbaum)
Newsgroups: comp.std.internat
Subject: Ukraine: Language tagging
Date: 8 Jan 93 13:16:17 GMT
Organization: University of Pittsburgh

>Proto-Latin, proto-Cyrillic etc sorting makes a lot of sense for
>business applications, providing uniformed way to deal with
>multilingual lists (and such sorting has an attractive quality --
>it "automatically" reduces to a national sorting after deletion
>of foreighn letters). Somehow, it is a simplified scheme of
>sorting large libraries use.

Multilingual sorting, even for languages that share a script, seems to
me to be first a conceptual, and only second an implementational,
problem. That is, if the soft sign sorts after <ja> in Ukrainian but
several letters before it in other Slavic languages written in Cyrillic,
what is _the correct_ pan-Cyrillic sorting order? Neither solution will
automatically reduce to correct sorting for all languages if the
Cyrillic inventory contains a single soft sign and a single <ja>. We
could create two soft signs, one for Ukrainian and one for everyone else
(similarly to the solution proposed for certain Latin letters with
diacritics), but then the "proto-Cyrillic" multilingual sort loses much
of its attractiveness. 

I see the primary problem as determining how we want multilingual lists
to sort, rather than how we want to implement the sorting. I don't see
any one solution as uniquely and obviously better than all others. The
problem is, for reasons I hope are obvious, even worse when a list mixes
scripts. As a frequent and frustrating real life example, how should I
sort a mixed Russian/English bibliography that cites both English and
Russian publications by a single author whose name begins with "V"? It
makes sense to separate the scripts, but it also makes sense to keep all
works by a single author in the same place. 

(It should be noted that a 1990 Ukrainian orthographic manual proposes
reordering the Ukrainian alphabet to conform to Russian and other
languages, so that the soft sign will no longer follow <ja>. How
comprehensively this is applied in practice remains to be determined.
Meanwhile, the problem is nonetheless real and general, viz. the similar
examples from Latin-alphabet orthographies.)

>Ukrain, say, use characters of both Cyrillic and Latin scripts, etc.

??? What Latin-script letters does Ukrainian use?

--David
--
-- 
Professor David J. Birnbaum         djbpitt+@pitt.edu [Internet]
The Royal York Apartments, #802     djbpitt@pittvms   [Bitnet]
3955 Bigelow Boulevard              voice: 1-412-687-4653
Pittsburgh, PA  15213  USA          fax:   1-412-624-9714
From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: Unicode ISO10646 CharacterEncoding
Date: 10 Jan 93 18:26:13 GMT
Article-I.D.: life.1ippo5INN7b3
Organization: MIT Artificial Intelligence Laboratory
NNTP-Posting-Host: wheat-chex.ai.mit.edu

In article <2675@titccy.cc.titech.ac.jp> mohta@necom830.cc.titech.ac.jp (Masataka Ohta) writes:
>On the other hand, one can't use process code for interchange, unless
>you are living in the closed world, because other applications won't
>accept it.

Oh, how about ASCII?  It makes a pretty decent process *and* interchange
code for English text.

>You should be crazy. If you are mapping a file under possibly networked
>environment (these days, all environments are so), you can't use multiple
>octet fixed width code because of endeaness. Don't say signature, because
>it makes everything complex and, thus, slow, and your reasoning to map
>files should be for efficiency and for simplicity.

Signatures are a far cry simpler than variable length encodings.

Glenn Adams
Newsgroups: comp.std.internat
From: boyd@prl.dec.com (Boyd Roberts)
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages
Organization: Digital Equipment Corporation - Paris Research Laboratory
Date: Fri, 8 Jan 1993 09:27:54 GMT

In article <1993Jan7.033153.12133@fcom.cc.utah.edu>, terry@cs.weber.edu (A Wizard of Earth C) writes:
> 
> Consider a newline terminated text database containing fixed length lines,
> or consider a database consisting of variant text records in fixed fields.
> In either case, the amount of data per field is now variant on Runic encoding.
> For instance, if we accept the Plan-9 soloution, an application used in
> both England and the US will vary as to how much data is representable per
> fixed field based on whether or not that data contains the English "#"
> character.  This gets worse the further you get from base ASCII coding.

Using Plan 9 utf you know the maximum size in bytes that a Rune can
be encoded into.  Fixed fields only have to be a multiple of this value,
defined to be to be UTFmax.  So where's the problem?

We can take this piece of code as working example (ie. functioning)
which takes an array of Runes, encodes them and then calls write
to send them in a reasonably efficient manner (ie 1 Rune != 1 write):

    void
    sendrunes(Rune *r, ulong len)
    {
        char    s[128 * UTFmax];

        while (len)
        {
            char    *p;

            for (p = s; p <= &s[sizeof s - UTFmax] && len; len--)
            {
                 int     n;

                 n = runetochar(p, r++);
                 p += n;
            }

            (void)write(fd, s, p - s);
        }
     }

Runes are your friend.


Boyd Roberts			boyd@prl.dec.com

``When the going gets wierd, the weird turn pro...''
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Language tagging
Date: 6 Jan 93 21:45:14 GMT
Organization: Tokyo Institute of Technology

In article <jenkinsj-050193090315@tseng.taligent.com>
	jenkinsj@blowfish.taligent.com (John H. Jenkins) writes:

>Looking at the question closely, however, I'm not sure that it's an
>issue of legibility so much as an issue of representability.  For
>example, suppose I have a plain [Unicode] text of a Japanese commentary
>on the Confucian classics.  Which parts should I display using a
>"Japanese" font and which using a "Chinese" font?
>
>The answer is:  You have to go through by hand and fix it.  (*blech*)

>Next scenario, however:  I download the King James Bible (say) from an
>ftp site in plaintext form.  The KJV uses italics extensively with a
>specific semantic.  Which parts should I display using italics?
>
>The answer:  You have to go through by hand and fix it.  (*blech*)

Very very fair comparison.

You have claimed that when italicizing some part of some plaintext, we
must do it by hand. Very interesting.

Then, consider the following senarios.

<I have a plain [ISO 2022] text of a Japanese commentary
<on the Confucian classics.  Which parts should I display using a
<"Japanese" font and which using a "Chinese" font?
<
<The answer is: As designated by the escape sequences.

						Masataka Ohta

PS

May the hell fire burn you for fixing the KJ Vible wrongly.
Newsgroups: comp.std.internat
From: christ@stc.nato.int (Brian Christiansen)
Subject: User interface
Organization: SHAPE Technical Centre, NL
Date: Thu, 7 Jan 1993 09:20:20 GMT

Could somebody provide me with pointers to guidelines, recommendations
or standards for the usage of colours, icons etc. in the man-machine
dialogue?

Regards,

Brian
-- 
B. Christiansen                                 christ@stc.nato.int 
SHAPE Technical Centre 
The Hague, The Netherlands                      +31 70 31 42 479 
-----------------------------------------------------------------------------
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Dumb Unicoders
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages
Date: 6 Jan 93 22:15:28 GMT
Organization: Tokyo Institute of Technology

In article <TaaXwB1w165w@blues.kk.sub.org>
	kosta@blues.kk.sub.org (Kosta Kostis) writes:

>Well, at least one that is "universal" enough from a Japanese point
>of view  ;-), but maybe you're right. So, just to satisfy my curiousity:
>how do you represent 18 or 20 bits with 8-bit octets?

As you want. But, according to Shanon, variable length encoding would
be better.

>Wouldn't be there be some "wasted" bits?

How does that matter? Aren't there unoccupied code points in Unicode?

>As I stated before, I think 16-bit UniCode is one step "aside" and one
>into the "right" direction. Does that sound like "fandom"?

Unicode is worse than ISO 2022 for its universalness.

>> But, as for reg-exp pattern, Vadim's suggestion is quite right.
>
>What do you use reg-exp patterns for?

To search some pattern in a large text file(s).

>Local versions for that will never go, I believe.

What we need is, of course, the universal version of reg-exp.

>PS: I am *not* a "UniCode"r, but I strongly feel the need for a universal
>    character set and I want it *implemented* in *this* century. :-)

Then, throwaway Unicode as soon as possible, as we have less than 8 years.

							Masataka Ohta
From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages
Date: 10 Jan 93 18:22:14 GMT
Article-I.D.: life.1ippgmINN7af
Organization: MIT Artificial Intelligence Laboratory
NNTP-Posting-Host: wheat-chex.ai.mit.edu

In article <2676@titccy.cc.titech.ac.jp> mohta@necom830.cc.titech.ac.jp (Masataka Ohta) writes:
>Different Characters with different stroke counts are mapped to a
>single code point in 10646/Unicode.

For those in the listening audience who don't read Hanzi or Kanji, I should
say that Ohta-san is correct, but in a completely frivolous way.  Stroke
counts are a bit subjective.  Some people count some elements as distinct
strokes, whereas others do not.  However, there is no systematic difference
in the way that Japanese count strokes versus the way that Chinese count
strokes versus the way that Koreans count strokes.  The ways that CJK
standards bodies count strokes are extremely uniform; therefore, the
unification of CJK characters into single Han code elements does not 
remove discriminating information; rather, it preserves the common, important
information (formal and functional) represented by the unified elements.

>Thus, if you use 10646/Unicode, stroke count is context dependent. Don't
>use it.

This is a bit like saying "Japanese (or English or another language) is
context dependent.  Don't use it."  Come on Ohta-san, return to reality.
Try making a rational argument.

Glenn Adams
From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Date: 9 Jan 1993 18:23:18 GMT
Organization: MIT Artificial Intelligence Laboratory
NNTP-Posting-Host: wheat-chex.ai.mit.edu
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages

In article <1993Jan9.031217.27425@fcom.cc.utah.edu> terry@cs.weber.edu (A Wizard of Earth C) writes:
>In article <1993Jan8.094119.6795@prl.dec.com> boyd@prl.dec.com (Boyd Roberts) writes:
>>Tell me how I sort on stroke count in Unicode without ``locale-specific
>>algorithms or hard-coded data''?

If it is Han characters that are being sorted, then Unicode already orders
them according to KangXi radicals (214) and (additional) stroke count.
Characters with simplified radicals immediately follow those with the
traditional radical.  Ties in these orders are broken by using the orders
of KangXi Zidian, Dai Kanwa Ziten, Hanyu Da Zidian, and Dae Jaweon in
this order.  To quote the relevant passage from the Unicode standard,
volume 2 (p. 14):

  "When a character is found in the KanXi Zidian, it follows the KangXi
  Zidian order.  When it is not found in the KangXi Zidian and is found in
  Dai Kanwa Ziten, it is given a position extrapolated from the KangXi
  position of the preceding character in Dai Kanwa Ziten.  When it is not
  found in either KangXi or Dai Kanwa, Hanyu Da Zidian and Dae Jaweon
  dictionaries are consulted in a similar manner."

Total stroke-count and four-corner (et al.) orderings are typically
used only for input mechanisms.  It is trivial to transform Unicode
Han encodings into any of various weighted indexes for sorting by using
a simple table lookup.

If anyone is interested in obtaining a Unicode Kanji Database,
I have one available for interested parties.  It contains the following
information:

  Unicode to JISX0208 Mappings
  Onyomi pronunciation(s) in katakana of each Unicode Kanji.
  Kunyomi pronunciation(s) in hiragana of each Unicode Kanji.
  Romaji for each on/kun kana reading (yomi).
  Radical, total stroke count, and added stroke count of each Unicode Kanji.
  
  *Only those Unicode Han characters which are also in JISX0208 are
   present in this database. 

Please send mail to <ucjis-request@metis.com> if you would like further
information.

Glenn Adams
From: ag129@cus.cam.ac.uk (Alasdair Grant)
Newsgroups: comp.std.internat
Subject: Re: ISO paper sizes
Date: 11 Jan 93 11:32:09 GMT
Organization: U of Cambridge, England


Perhaps the reason for the confusion might be that the article on HyTime
which I saw (and which I've completely failed to find again) used "NIST
inch" to distinguish it from the traditional US inch.  This would have been
very obscure to non-US readers, but that is not unusual.  It would fit with
NIST having standardised on the SI-derived inch, and with HyTime not having
any unit representing 1/39.37 m.  Just a guess.