HDTenEightyP/NetNews huggingface dataset:
From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.std.internat
Subject: Re: Radicals Instead of Characters
Date: 25 Jan 93 22:21:07 GMT
Article-I.D.: life.1k1p4kINNjse
Organization: MIT Artificial Intelligence Laboratory
NNTP-Posting-Host: wheat-chex.ai.mit.edu

In article <1993Jan25.194330.680@ifi.unizh.ch> mduerst@ifi.unizh.ch (Martin J. Duerst) writes:
>
>In article <1jtbfvINNqvr@life.ai.mit.edu>, glenn@wheat-chex.ai.mit.edu (Glenn A. Adams) writes:
>>   precomposed encoding
>> 
>>   2^20 precomposed characters *
>>   2^16 bits/precomposed character =
>>   2^36 bits
>> 
>>   decomposed encoding
>> 
>>   2^20 precomposed characters *
>>   2^4  decomposed characters/precomposed character *
>>   2^11 bits/decomposed character =
>>   2^35 bits

Sorry, my arithmetic was broken, I should have had:

precomposed encoding

 2^20 precomposed characters *
 16   bits/precomposed character =
 2^24 bits = 2 MB

decomposed encoding

 2^20 precomposed characters *
 9    decomposed characters/precomposed character *
 11   bits/decomposed character =
 1.55 * 2^26 bits = 12.375 MB

This works out to a 6.2 times increase in size for the decomposed form.  If
the decomposition required fewer elements on average, say 4 instead of 9,
then that would reduce the expansion to 2.75 times the precomposed form.

I don't know if 4 element decompositions are possible on the average,
particularly given size and position ambiguities.  Even if it were
possible, not only would you require nearly 3 times the size as the
precomposed form, but you also would have the complexities introduced by
variable length text element (in this case, Han graphic symbol = Han
character) encodings.

The only reason I could see in pursuing this is to support encoding rarely
used characters rather than wasting the encoding space on little used
elements.  If all characters through the 90 percentile are encoded as
single code elements and the remainder decomposed using a decomposition
scheme, then I expect the best efficiency will hold on both size and
on encoding space usage.  Text processing algorithms will be somewhat
complicated by having to deal with variable length text element encodings;
however, the same is true even now for dealing with Unicode non-spacing
diacritics on Latin and Cyrillic base forms.

One way out of this last problem is to use the Unicode Private Use Zone
(6144 code positions) as a dynamically reconfigurable character set area.
Then, upon input, translate rare decomposed Han elements (or other decomposed
text elements) to a code point which is dynamically assigned from the private
use zone.  Of course, when interchanging the data, it would have to be
retranslated back to the decomposed form -- I assume you  don't want to send
someone Unicode data with private use zone data, that is, unless you are
in complete control of both sending and receiving parties.

Glenn Adams
Newsgroups: comp.std.internat
From: johnt@unipalm.co.uk (John Taylor)
Subject: Where to get ISO standards
Keywords: ISO standards
Organization: Unipalm Ltd., 216 Cambridge Science Park, Cambridge CB4 4WA, UK
Date: Tue, 26 Jan 1993 12:36:44 GMT

Can someone please tell me where I can get copies of ISO standards ?
Preferably via FTP.

I am most interested in ISO-646 and ISO-8859.

Thank you
John Taylor
johnt@unipalm.co.uk
From: kosta@blues.kk.sub.org (Kosta Kostis)
Newsgroups: comp.std.internat
Subject: Re: Cleanicode
Date: Thu, 21 Jan 93 20:54:17 MET
Organization: The Blues Family
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

djbpitt+@pitt.edu (David J Birnbaum) writes:

> In summation, it is clear to me that LGC should not be unified, but I
> would feel more comfortable if I could articulate what it means to be
> an autonomous script. Suggestions welcome, as always.

Greek, Latin and Cyrillic are three different *alphabets*.

When I say "alphabets" I'm referring to a small number of characters with
an order that allow the "user" of such an alphabet to write text in their
language. At least some characters in the above mentioned alphabets look
different and have different meanings or can't be found in all alphabets.

I see no need/benefit to unify different alphabets.

Additionally the ease of conversion (from ISO 8859-x) is important, too.

        Kosta


-- 
  Kosta Kostis, Talstrasse 25, D-6074 Roedermark 3, Germany
  kosta@blues.kk.sub.org                                        (home)
  please support ISO 8859-x & MIME!            = aeoeueAEOEUEss
From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Date: 23 Jan 1993 17:52:35 GMT
Organization: MIT Artificial Intelligence Laboratory
NNTP-Posting-Host: wheat-chex.ai.mit.edu

In article <ISHIKAWA.93Jan22211810@ds5200.personal-media.co.jp> ishikawa@personal-media.co.jp writes:

[Regarding 3 vs. 4 stroke grass radical...]

>I don't care how many strokes there are. BUT, for once, I STRONGLY
>DISAGREE.  If you ask ordinary Japanese of modern Japan in the street,
>they would say that the characters using the radical mentioned in the
>upper position above and the characters using the radical mentioned in
>the lower position above are CERTAINLY DIFFERENT. I would bet about
>1/3 to half the people would even say, "you are writing an INCORRECT
>characters."!

That they are judged different is not the issue; the issue is whether
a person can read the text (i.e., is it legibile).  Even Japanese who
would judge it incorrect could still read it, and, yes, they *may*
notice it, surely they would in a typeset document, but would they notice
it in a 14-point screen font?

Perhaps I grew up being trained that a dollar sign always had a vertical
line extending completely through the <S>.  If I saw a Courier dollar
sign <$> [I'm using a Courier font here], I might say:  this is an
INCORRECT character.  But I would not misunderstand it!

The point is that there is no situation in Japanese where having a
3 or 4 stroke grass radical makes a difference in meaning.  From a
linguistic perspective, these would be called "allographs" of a single
"grapheme".  A collection of allographs of a grapheme are those symbols
which instantiate the grapheme and which, if any other allograph is
substituted, does not create a difference in meaning.  For example,
the grapheme <$> has a number of allographs, one with one vertical
stroke, one with two, intersecting or non-intersecting vertical strokes;
lower case <a> and <g> have two common allographs each, etc.

If you can show me a text where the difference between a 3 and 4 stroke
grass radical makes a difference in meaning, then I may agree with you.
[I would exclude from this test a text which is artifically making a
distinction, e.g., to create a pun based on the written form of the
character.]

Your arguments on "correctness" are based on aesthetic criteria, and
not on meaningful distinction.  The Han unification makes only the
latter the criteria for distinctness of abstract form; the notion of
form "abstraction" is simply the removal of aesthetic criteria.

If you must maintain aesthetic distinctions between different forms of
a single unified Han character, then you must use some form of rich
text, e.g., escape sequences, out-of-band style data, or whatever it
takes to allow your system to display that single character differently
depending on its linguistic (or font) context.

Glenn Adams
From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.std.internat
Subject: Re: European characters (was 8-bit news)
Date: 23 Jan 1993 17:02:28 GMT
Organization: MIT Artificial Intelligence Laboratory
NNTP-Posting-Host: wheat-chex.ai.mit.edu

In article <1993Jan22.122553.4823W@lumina.edb.tih.no> ketil@edb.tih.no (Ketil Albertsen,TIH) writes:
>10646/1 is supposedly a 32-bit identification scheme intended to
>cover just about any printable symbol in the world, while 10646/2, or Unicode,
>is a subset using 16 bit codes for text symbols (characters or Asian-
>style symbols).

ISO/IEC 10646-1:1993 (to be published in the first or second quarter this
year) defines one repertoire of characters and two encoding forms
The encoding forms are called UCS4 (Universal Character Set 4-Octet)
and UCS2 (Universal Character Set 2-Octet), the former interpreted as 32-bit
unsigned integers, the latter as 16-bit unsigned integers.  The UCS2 encoding
form is related to the UCS4 encoding form by zero extension; that is, by
zero extending the 16-bit form to 32 bits, the equivalent UCS4 encoding form
is created.

No characters are currently assigned to codepoints (bit combinations) outside
of UCS2 (also called the Basic Multilingual Plane or BMP for short).  Out
of the 65,536 distinct bit combinations in UCS2, 34,168 are assigned to
characters, 6,467 are reserved, and 24,901 are available for future assignment.

The UCS2 encoding space is divided into sections, whose contents are
characterized below:

A-Zone (Alphabetic)	(11,892 assigned, 65 reserved, 8,011 available)

  Alphabets
  Hangul Jamo Alphabet
  Latin & Greek Precombined Forms
  Symbols
  CJK Auxiliaries
  Hangul Precombined Syllable Forms

I-Zone (Ideographic)	(20,902 assigned, 0 reserved, 90 available)

  CJK Unified Ideographs

O-Zone (Open)		(0 assigned, 0 reserved, 16,384 available)

  Unassigned

R-Zone (Restricted)	(1,374 assigned, 6402 reserved, 416 available)

  Private Use Area
  CJK Ideograph Compatibility
  Presentation Compatibility
  Other Compatibility & Specials

Glenn Adams
From: prc@uk.co.gec-mrc
Newsgroups: comp.std.internat
Subject: ISO standard for icons?
Date: 25 Jan 93 09:38:45 GMT
Reply-To: prc@uk.co.gec-mrc ()
Organization: GEC-Marconi Research Centre, Great Baddow, Essex

I'm sure I saw somewhere recently that there is an ISO standard for
icons available. Could someone please tell me the standard number.

Alternatively, is there an on-line list of the ISO standards available
anywhere? I realise that I would have to pay for the standard itself,
but a list of available standards with brief descriptions of what they
cover would be very useful.

Thanks in advance,

Pete C.
--
Pete Chalmers               | Tel: +44 245 73331 x 3243
GEC-Marconi Research Centre | Fax: +44 245 75244
Chelmsford, CM2 8HN      UK | Internet   : prc@gec-mrc.co.uk
Newsgroups: comp.std.internat,soc.culture.nordic,soc.culture.german,soc.culture.french,alt.folklore.computers,news.future,news.admin.misc
From: ddawson@rodan.acs.syr.EDU (Dick Dawson)
Subject: Re: European characters (was 8-bit news)
Date: Thu, 21 Jan 93 00:56:56 EST

>Further problem: I haven't found any way to get to any useful code
>pages from any application, esp where needed, eg: in a text editor. 
>Sure, I can re install moststupiddos for another country.  Therefore
>trashing my whole file system.  What I need is a switch on the fly
>system, or, better, European characters available at any time by some
>simple key combination.
>
>Any help available?
>
>No moststupidwindows please.

Thanks, all around...   Since I posted this request I've gotten
several helpful responses.

In msdos 5.0, there is a file KEYB.COM that will invoke the keyboard
layout and appropriate code page for one country other than the
installed country.  <alt><F1> and <alt><F2> will toggle the two setups
while in Wordstar, at least.  This might not be available in
Wordperfect as it already assigns <alt><Fn> to other functions.  This
begins to resolve some of my needs although there are bugs.  The
settings for Norway and for Denmark to not yield o-slash or O-slash;
they provide a c with broken verticle bar (cents sign) and a Y with
double cross bars, probably a yen sign.  This might be a glitch in my
video card.  But at least I've got a start.  Thanks again for the
help.

Another interesting bit:  Some news postings have various punctuations
in the places of Scandinavian characters that do not exist in the
American alphabet.  I don't remember the equivalences.  They always
show as the puncutations, curly braces, vertical bar, etc, on my
terminals and on the Xerox 4075 laser printer I usually use.  No
surprise, as the data links at Syracuse Univ. are all 7bit.  Except,
apparently to a HP Laserjet/Postscript printer... where the actual
Scandinavian characters appear, about half of the time.  I'm guessing:
some writers are substituting US punctuations and probably are on
machines that can't generate Scandinavian characters but others are
apparently using machines that actually do create Scandinavian
characters and are sending them in postings; they appear as US
punctuations when the high bit is stripped.

I'm still after the ISO 16bit (I hope) _standard_.

Dick   ddawson@mailbox.syr.edu
From: melby@dove.yk.fujitsu.co.jp (John B. Melby)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Date: 22 Jan 93 16:05:02 GMT
	<2770@titccy.cc.titech.ac.jp> <1jnlg0INN9n4@life.ai.mit.edu>
Organization: Open Systems Group, Fujitsu Limited, Yokohama
In-reply-to: glenn@wheat-chex.ai.mit.edu's message of 22 Jan 93 02:17:36 GMT

>[stroke count] is like having a dollar sign
>glyph with one stroke or two strokes through an S, or having one with
>a single vertical stroke that intersects the whole S, or extends vertically
>from the top and bottom arches without fully intersecting the S.

Stroke count, per se, is more of a handwriting issue.  Stroke position is
more important.

An example of a unification to which people have very strong objections
is that of the Japanese and Chinese simplified forms for "yo (ataeru) / yu4"
("to impart," "to").  The Chinese form has a horizontal stroke that does not
intersect the main stroke, whereas the horizontal stroke in the Japanese form
does intersect it.

The difference above is more serious than the dollar-sign variants
mentioned previously, since official modern simplified forms of
characters are more well-defined (and enforced) than traditional
variants.  (Food for thought:  have you heard of a US student
being criticized by a teacher for having the wrong number of bars
in a dollar sign?)

-----
John B. Melby
Fujitsu Limited, Yokohama
melby@yk.fujitsu.co.jp
Newsgroups: comp.std.internat
From: clive@x.co.uk (Clive Feather)
Subject: Re: ISO paper sizes
Organization: IXI Limited
Date: Mon, 25 Jan 1993 18:29:53 GMT

In article <8582@charon.cwi.nl> dik@cwi.nl (Dik T. Winter) writes:
> Quoting from my basic reference on units %, subject yard:
>	"... In England until 1963 the Imperial Standard Yard was the legal
>	unit of length.  It was defined in the Weights and Measures Act of
>	1878 as the distance between two points on a specified metal bar.
>	In 1963 the yard was redefined as being equal to 0.9144 metres
>	exactly.
> --
> %  H.G.Jerrard & D.B.McNeill, A Dictionary of Scientific Units.  Science
>    Paperbacks, Chapman & Hall Ltd., London, 1966.

Your basic reference is wrong. The Weights and Measures Act 1948 [not
"of 1948"] defined the yard to be 3600/3937 of the metre, whose
definition I forget. The 1963 [?, I thought it was 1967] Act redefined
it to 0.9144 metres, and defined the metre using the same wording as the
SI definition. Later Acts have kept the definition of the metre in step
with SI.

-- 
Clive D.W. Feather     | IXI Limited         | If you lie to the compiler,
clive@x.co.uk          | Vision Park         | it will get its revenge.
Phone: +44 223 236 555 | Cambridge   CB4 4RZ |   - Henry Spencer
Fax:   +44 223 236 550 | United Kingdom      |
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Cleanicode
Date: 24 Jan 93 12:25:19 GMT
Organization: Tokyo Institute of Technology

In article <75sqXB1w165w@blues.kk.sub.org>
	kosta@blues.kk.sub.org (Kosta Kostis) writes:

>Greek, Latin and Cyrillic are three different *alphabets*.
>
>When I say "alphabets" I'm referring to a small number of characters with

I can't understand why it must be "a small number". Anyway,

>an order that allow the "user" of such an alphabet to write text in their
>language. At least some characters in the above mentioned alphabets look
>different and have different meanings or can't be found in all alphabets.

German alphabet has a character "A with umlaut" which can't be found in
French alphabet.

They are, by your definition, different alphabets.

Why you unify German alphabet and French alphabet, then?

>Additionally the ease of conversion (from ISO 8859-x) is important, too.

Of course, the ease off conversion from ISO 2022 conformant text
using all of GB, JIS and KCS is is also important.

						Masataka Ohta
From: kosta@blues.kk.sub.org (Kosta Kostis)
Newsgroups: comp.std.internat
Subject: behaviour (was Dumb Americans)
Summary: there are more than enough wars - please don't start a new one
Keywords: flames
Date: Mon, 25 Jan 93 02:16:52 MET
Organization: The Blues Family
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

mohta@necom830.cc.titech.ac.jp (Masataka Ohta) writes:

Sorry, this person doesn't seem to read mails... ;-)

> Glenn, you stop saying anything on Han. You know nothing.

Masataka Ohta, have you ever noticed World War II is over?  ;-)

Instead of insulting people in public, which is annoying even to the
people who only witness your social incompetence, you should start to
discuss in a polite way in a public international medium. As long as
you don't behave yourself you're to be ignored - no matter how good
or bad your arguments are.

Some people might also come to the conclusion that you have never read
the Netiquette and mail you a copy...

> Meaning is meaningless here. You can write Greek script with Latin alphabet
> without changing meaning.

You obviously don't know much about European scripts, languages, cultures,
religions, history etc., so why do you throw stones in a glass house?  :-(

	Kosta

-- 
  Kosta Kostis, Talstrasse 25, D-6074 Roedermark 3, Germany
  kosta@blues.kk.sub.org                                        (home)
  please support ISO 8859-x & MIME!            = aeoeueAEOEUEss
From: wbdst+@pitt.edu (William B Dwinnell)
Newsgroups: comp.std.internat,soc.culture.nordic,soc.culture.german,soc.culture.french,alt.folklore.computers,news.future,news.admin.misc
Subject: Re: European characters (was 8-bit news)
Date: 21 Jan 93 13:28:16 GMT
Followup-To: comp.std.internat
Organization: University of Pittsburgh


Whatever happened to Unicode?
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages
Date: 21 Jan 93 17:20:13 GMT
Organization: Tokyo Institute of Technology

In article <1ippgmINN7af@life.ai.mit.edu>
	glenn@wheat-chex.ai.mit.edu (Glenn A. Adams) writes:

>>Different Characters with different stroke counts are mapped to a
>>single code point in 10646/Unicode.
>
>For those in the listening audience who don't read Hanzi or Kanji, I should
>say that Ohta-san is correct, but in a completely frivolous way.  Stroke
>counts are a bit subjective.  Some people count some elements as distinct
>strokes, whereas others do not.

Wrong. That's not what I'm referring.

>However, there is no systematic difference
>in the way that Japanese count strokes versus the way that Chinese count
>strokes versus the way that Koreans count strokes.  The ways that CJK
>standards bodies count strokes are extremely uniform;

True, but it is misleading.

Consider, for example, a very common radical used for plant-related
characters, such as grass and flowers (character code from 827C to 864F
of DIS 10646-1.2).

Its shape in Japanese or mainland China is

        *        *
     ****************
        *        *

whose stroke count is 3, while its shape in Taiwan and Korea is

        *        *
     *******  *******
        *        *

whose stroke count is 4. But, these different characters with differently
shaped radicals are unified in Unicode.

>therefore, the
>unification of CJK characters into single Han code elements does not 
>remove discriminating information;

Therefore, the stroke count information, which is "THE EXTREMELY UNIFORM"
property of characters, are destroyed by Han unificaiton.

This is the reality.

>This is a bit like saying "Japanese (or English or another language) is
>context dependent.  Don't use it."  Come on Ohta-san, return to reality.
>Try making a rational argument.

I have come again. I'll be happy if you stop dreaming and shut up.

						Masataka Ohta
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Radicals Instead of Characters
Date: 22 Jan 93 09:44:54 GMT
Organization: Tokyo Institute of Technology

In article <1jfgq1INNqmn@flop.ENGR.ORST.EDU>
	crowl@jade.CS.ORST.EDU (Lawrence Crowl) writes:

>The question I was asking was "can you _identify_ a han/kanji character
>based on a sequence of radicals"

No, you can't. Radicals are for indexing only. The rest of the character
has its own complex shape.

>and "would it be reasonable to encode
>han/kanji on that basis".

Such encoding is too lengthy.

>Agreed.  However, there is no natural size for tables.  Table sized of
>4000 are much cheaper than table sizes of 64000.

If you use radical based encoding, it makes everything complex.

Moreover, you will have to have sixteen 4000 entry tables which is as
large as a single 64000 entry table.

>But, can sixteen bits represent _all_ historical Han characters _and_
>the historical texts of all other languages?  My guess is 16 bits can
>_if_ Han characters are coded as radicals,

Maybe nor may not be. Many complex Han characters are just unique.

>If the level 1 Han characters were also coded as radicals where
>possible, you'd have a coding system like what I was proposing.  Of
>course, the charactes might be several radicals long.

BTW, from the view point of programmers, combining characters are
just unusable.

						Masataka Ohta
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.std.internat
Subject: Re: Radicals Instead of Characters
Date: 24 Jan 93 12:19:05 GMT
Organization: Tokyo Institute of Technology

In article <1jpj9sINNlie@flop.ENGR.ORST.EDU>
	crowl@jade.CS.ORST.EDU (Lawrence Crowl) writes:

>>>The question I was asking was "can you _identify_ a han/kanji character
>>>based on a sequence of radicals"
>>
>>No, you can't. Radicals are for indexing only. The rest of the character
>>has its own complex shape.
>
>If you can use radicals for indexing, then you can use them to identify
>characters.

No you can't. The correspondence is not one to one. It is many to many.

>>Such encoding is too lengthy.

>An encoding every variant of every character ever written is not?

Of course, not.

>Your non-unified
>approach would require roughly eighteen bits per character.

What's the problem? 18 bit character set is just fine.

>>Moreover, you will have to have sixteen 4000 entry tables which is as
>>large as a single 64000 entry table.
>
>No, I don't have to have sixteen 4000 entry tables.  I only need one.

Could you please elaborate?  Your argument leaves me unconvinced.

>>If you use radical based encoding, it makes everything complex.

>Could you please elaborate?  Your argument leaves me unconvinced.

Perhaps because you don't understand that you need sixteen 4000 entry
tables.

>>>But, can sixteen bits represent _all_ historical Han characters _and_
>>>the historical texts of all other languages?  My guess is 16 bits can
>>>_if_ Han characters are coded as radicals,
>>
>>Maybe nor may not be. Many complex Han characters are just unique.
>
>Unique in what sense?  Examples?

As those characters are really complex I don't write figures of them.
See DIS 10646-1.2.

>>BTW, from the view point of programmers, combining characters are
>>just unusable.
>
>I am a programmer.

Then, how can your program detect the character boudaries?

That you can't find character boudaries without lookaheading makes
all programs complex and interractive programming impossible.

Are you really a programmer?

						Masataka Ohta