HDTenEightyP/NetNews huggingface dataset:
From: sommar@enea.se (Erland Sommarskog)
Newsgroups: comp.std.internat
Subject: Re: Language tagging
Date: 2 Jan 93 23:17:03 GMT
Organization: Enea Data AB

David J Birnbaum (djbpitt+@pitt.edu) writes:
>I think the objections to the input-related problems of Vadim's proposal
>are misdirected, since both Vadim's proposal and Unicode require the
>user to input language identifying information during data entry if this
>information will be needed for later processing. Under Vadim's
>proposal, what would be input would be an instruction (not part of the
>stored text stream) to shift to the appropriate subset of characters.
>Under a system built on a Unicode character set, what would be input
>would be some sort of language or locale tagging that would be entered
>into the text at a higher level than character set.
>
>In both cases, if you want language-specific data in your text stream,
>you have to say so during input. If I need to insert Bulgarian words
>into a Russian text stream I can do so without indicating a change,
>as long as I understand that the consequence will be that the Bulgarian
>data will be treated like Russian.

Then I have to ask you explain something about Unicode I don't
know. It is true that if you are using language-dependent features
such as spell-checking and hyphenation, while inputting the text,
then you have to know what you are doing with Unicode. But once
you're done with it, it doesn't matter any longer, until the next 
time you want to process the text in some way. With Unicode you
can decide whether to treat the text in Bulgarian or Russian,
with Vadim's system you're stuck unless you convert it. (Well,
maybe you'll get away with Bulgarian and Russian, but not with
Swedish and German. Or as I demonstrated in another article,
Swedish and Danish.)

But of course, if I have CCCP in a Swedish text and copies the word
to a Russian text I will see funny things both with Vadim Antonov's
system and 10646. (And I am sure this confusion will cost someone
a couple of wasted work hours in finding the error.) But at least
shifting scripts is a more obvious, than changing languages. If I 
switch from Swedish to Russian I would probably change the keyboard 
set-up, but not if I switch from Swedish to German - there is no 
reason to.
-- 
Erland Sommarskog - ENEA Data, Stockholm - sommar@enea.se
Jag gav en k{ck tjeck en check.
From: emv@msen.com (Edward Vielmetti)
Newsgroups: comp.std.internat
Subject: MIME FAQ available, input sought
Date: 4 Jan 1993 00:54:46 GMT
Organization: Msen, Inc. -- Ann Arbor, Michigan
NNTP-Posting-Host: garnet.msen.com
X-Newsreader: TIN [version 1.1 PL8]

I have a draft of an FAQ on MIME, the Internet multimedia mail format.
It was posted to comp.mail.mime, but if you missed it there or would
like teh current draft please drop me a line. 

I have two parts of the FAQ I'd like to fill out with answers.

First is the internationalization part - if you have details or
experience or working software that properly handles any language
other than net.english, please mail me details.  Preferably or
ideally mail me details in a MIME message in your native language,
with all character sets tested so far as they go.

Second is the interaction of MIME and netnews.  I have heard rumblings
of MIME support in both INN and C news, and have seen code that
glues MIME support into GNUS.  If you have projects or working code
please let me know.  If there's language for son-of-1036 in the
works that would be useful too.  Of particular interest is code
that copes gracefully with the existing news systems that aren't
net.english, ie. fj.*, relcom.*, sfnet.* in a way that's consistent
with MIME encodings.

thanks muchly,

  Edward Vielmetti, vice president for research, Msen Inc. emv@Msen.com
        Msen Inc., 628 Brooks, Ann Arbor MI  48103 +1 313 998 GLOB

Newsgroups: comp.std.internat
From: peter@sug.org (Peter Salus)
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages
Organization: The Sun User Group
Date: Thu, 31 Dec 1992 15:06:02 GMT

In article <1992Dec31.110735.17551@infodev.cam.ac.uk> ag129@cus.cam.ac.uk (Alasdair Grant) writes:
>In article <8490@charon.cwi.nl> dik@cwi.nl (Dik T. Winter) writes:
>But 'locale' is not just country; in the locale of names in telephone
>directories, 'Mac<uppercase>' sorts with 'Mc<uppercase>', 'St.' is expanded
>to 'Saint' etc.  

Yes, indded.  And this leads to such brilliant expansions of
[in Boston] St. Botolph St. to Saint Botolph Saint or 
Street Botolph Saint, etc.

Peter

-- 
---------------------------------------------------------------------

Sun User Group, Inc; Suite 315; 1330 Beacon St.; Brookline, MA 02146
	voice +1 617 232-0514		fax +1 617 232-1347
Newsgroups: comp.std.internat
From: boyd@prl.dec.com (Boyd Roberts)
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: ISO10646 Unicode
Organization: Digital Equipment Corporation - Paris Research Laboratory
Date: Fri, 1 Jan 1993 16:39:27 GMT

In article <1i13rrINNars@rodan.UU.NET>, avg@rodan.UU.NET (Vadim Antonov) writes:
> 
> We were talking about lexicographical sorting, not abouth phonetics.
> Your argument is irrelevant.
> 

In Chinese it would be natural if `sort' could sort on stroke count.
Chinese dictionaries have indexes based on stroke count, as there is
really no other way of locating a glyph whose reading (pronounciation)
you do not know.

In Japanese the problem is similar, but you have the kana which helps.

And you say that all this `side information' should be encoded into the
character set?  Just how big will your encoding be?


Boyd Roberts			boyd@prl.dec.com

``When the going gets wierd, the weird turn pro...''
From: torbenm@diku.dk (Torben AEgidius Mogensen)
Newsgroups: comp.std.internat
Subject: Re: Week Numbering
Date: 21 Dec 92 17:26:52 GMT
Organization: Department of Computer Science, U of Copenhagen

plate@ruc.dk (John Plate) writes:

>In Europe (?) the first week of a year is the week that has more than 
>3 days in it. Weeks goes from monday to sunday.

>If a year starts with a week with less than 4 days, it is week 53 of
>the previous year.

or it could be week 52. This will happen in 1994.

	Torben Mogensen (torbenm@diku.dk)
Xref: sparky comp.std.internat:1007 soc.culture.nordic:8080 soc.culture.german:9586 soc.culture.french:9385
From: kosta@blues.kk.sub.org (Kosta Kostis)
Newsgroups: comp.std.internat,soc.culture.nordic,soc.culture.german,soc.culture.french,alt.f
Subject: Re: 8-bit news
Summary: What's your original problem?
Keywords: CodePages DOS
Date: Sun, 03 Jan 93 22:28:16 MET
Organization: The Blues Family

ddawson@mothra.syr.EDU (Dick Dawson) writes:

> Further problem: I haven't found any way to get to any useful code
> pages from any application, esp where needed, eg: in a text editor. 

"Get" Code Pages? You can use *one* Code Page at a given time.
I don't know a single DOS-application that supports multiple
Code Pages in one document. Neither does DOS.

> Sure, I can re install moststupiddos for another country.  Therefore
> trashing my whole file system.  

What makes you believe that?
Have you ever played with your CONFIG.SYS: COUNTRY, DISPLAY.SYS, KEYB?

>                            What I need is a switch on the fly
> system, or, better, European characters available at any time by some
> simple key combination.

Code Page 850 is a good starter - even for the US. I use it right now,
but when my message gets transfered out of my system I translate it into
ISO 8859-1, so that's how the umlauts are encoded in my .signature.

> Any help available?
> 
> No moststupidwindows please.

Sad.  :-)  Windows allows more than one font and character set and
depending on what language version of Windows you use you may have
ISO 8859-1 as the default character set.

Windows NT will have UniCode (ISO 10646/16-bit).

	Kosta


-- 
  Kosta Kostis, Talstrasse 25, D-6074 Roedermark 3, Germany
  kosta@blues.kk.sub.org                                        (home)
  sw authors: please support ISO 8859-x!       = aeoeueAEOEUEss
Newsgroups: comp.std.internat
From: peter@ferranti.com (peter da silva)
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages
Organization: Xenix Support, FICC
Date: Fri, 1 Jan 1993 23:01:09 GMT

In article <8492@charon.cwi.nl> dik@cwi.nl (Dik T. Winter) writes:
> (Moreover, when sorting names I would prefer to sort C. van der Bilt
> under V if it is an American and under B if it is a Dutchman ;-).)

How about "da Silva"? I've found myself sorted as if my surname was "Da"
(before Daniels), "D A Silva" (Before D.A.T.), "Dasilva" (before Davies),
"da*" (after Z), and "Da\240silva" (after Day).

Given what sorting is *for*, I'd say sort using whatever rules the
person *reading* the list needs. These days, that means simple character
set order since the "person" is a program.
-- 
Peter da Silva                                            `-_-'
Ferranti International Controls Corporation                'U` 
Sugar Land, TX  77487-5012 USA
+1 713 274 5180                            "Zure otsoa besarkatu al duzu gaur?"
From: sommar@enea.se (Erland Sommarskog)
Newsgroups: comp.std.internat
Subject: Re: Language tagging
Date: 3 Jan 93 20:30:17 GMT
Organization: Enea Data AB

David J Birnbaum (djbpitt+@pitt.edu) writes:
>Concerning the former, I would normally require that my texts include
>language identification, so that the same "this is Bulgarian" or "this
>is Russian" information would be present in both a Unicode-based system
>and Vadim's system, although it would not be an inalienable part of the
>character set in the former. Thus, I would be "stuck with" language
>information under both systems. While Unicode is capable of
>representing text without language information (by eschewing the use of
>tags), I can't think of a situation where I would want to do so.

This I don't understand. If you throw away a short e-mail message
to a friend, why would it be necessary to tag the text with the 
language?

And why would "this is Bulgarian" be different from "this is a text
about motorcycles"?

I don't doubt that there are many situations you want to keep track
of which language a text is written in, but what I question is that
the language should is the only such important property. Peter Da
Silva discussed SGML tags in another article, and intuitively it seems
to me that SGML tags is precisely what you are looking for.

It is also worth noting that while plain Unicode you can send away
an untagged e-mail without problems, you cannot doing so with Vadim
Antonov suggests as far as I understand. If all you have is a number
of small 8-bit sets, then you must tag the text with the set being
used if you expect the receptor's default set being another than
the one you use, else the text may come up as garbage.
-- 
Erland Sommarskog - ENEA Data, Stockholm - sommar@enea.se
Jag gav en k{ck tjeck en check.
From: andrew@alice.att.com (Andrew Hume)
Newsgroups: comp.std.internat
Subject: Re: ISO 8859-X What are the X's?
Summary: a start
Keywords: ISO 8859
Date: 30 Dec 92 03:17:50 GMT
Article-I.D.: alice.24509
Organization: AT&T Bell Laboratories, Murray Hill NJ

In article <1739@airgun.wg.waii.com>, fvance@airgun.wg.waii.com (Frank Vance) writes:
> Can someone provide a brief overview of what the various parts of ISO
> 8859 are?  By this I mean, if 8859-1 is "Western European" and
> -8 is "Hebrew" (as I have seen them called), what are the other parts?
> -- 
> Frank Vance  +1.713.963.2426		Western Geophysical
> fvance@airgun.wg.waii.com		10001 Richmond Avenue


	there is no convenient quick description; each standard has in it
a list of languages that can written using the characters therein.
the summaries given by plan 9's tcs command are:

	8859-1	Latin-1 (Central European)
	8859-2	Latin-2 (Czech .. Slovak)
	8859-3	Latin-3 (Dutch .. Turkish)
	8859-4	Latin-4 (Scandinavian)
	8859-5	Part 5 (Cyrillic)
	8859-6	Part 6 (Arabic)
	8859-7	Part 7 (Greek)
	8859-8	Part 8 (Hebrew)
	8859-9	Latin-5 (Finnish .. Portuguese)
Xref: sparky comp.std.internat:931 news.admin.misc:916
From: andrew@alice.att.com (Andrew Hume)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Re: 8-bit news
Summary: yes, but its broken
Date: 26 Dec 92 03:43:25 GMT
Article-I.D.: alice.24483
Organization: AT&T Bell Laboratories, Murray Hill NJ

In article <1hg2afINNi81@nigel.msen.com>, emv@msen.com (Edward Vielmetti) writes:
> Chinese and Japanese already have several encodings into character
> sets that can be sent around through netnews - the fj.* news hierarchy
> carries quite a bit of stuff in it right now.  As it happens the fj
> groups chose a 7 bit encoding, but that doesn't remove the need for
> extra tools (from a Latin alphabet point of view) to read them.

	yep, and on every machine i can get at the fj.* news stuff,
the bloody escapes (033) have been stripped out, rendering the
SJIS encodings utterly useless.

			andrew hume
From: ag129@cus.cam.ac.uk (Alasdair Grant)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Keywords: Han Kanji Katakana Hirugana ISO10646 Unicode Codepages
Date: 31 Dec 92 11:07:35 GMT
Organization: U of Cambridge, England

In article <8490@charon.cwi.nl> dik@cwi.nl (Dik T. Winter) writes:
>Wrong Vadim.  You cannot even do it in the European languages.  You cannot
>even do it in German.  How would you assign codes such that the German
>A-umlaut sorts as if it is the letter combination AE, and at the same

But 'locale' is not just country; in the locale of names in telephone
directories, 'Mac<uppercase>' sorts with 'Mc<uppercase>', 'St.' is expanded
to 'Saint' etc.  You nearly always need to take into account other rules
than the character codes; therefore, if the character codes aren't right
for every language it really doesn't matter.
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Organization: I.E.C.C.
Date: 1 Jan 93 17:57:35 EST (Fri)
From: johnl@iecc.cambridge.ma.us (John R. Levine)

>it is my opinion that there is no way to make a simple character code that
>will perform sorting and character conversion automatically.

Having done my share of i18n, I thoroughly agree.  When I was writing the
international scaffolding for Javelin, a PC time-series modelling package,
we came up with a locale-like thing that let you load a country
configuration file.  The config file set the collating sequence, including
which characters sort together, and whether there are pairs of characters
that sort as one like spanish ch and ll, or single characters that sort as
two, like the German umlauted vowels.  It also loaded the strings that
were inserted automatically into graphs and printouts, e.g. month names
and words like "Millions."

What did not change was the message strings in the program or the table of
function and macro names, all of which were version-specific.  That is, if
you bought a French version of Javelin, it always spoke French, but you
could load in a country driver to produce reports in German or Spanish or
Dutch.  The separation between the "locale" for the program and the
"locale" for the reports was quite useful, and the users liked it.  The
program message strings were linked in when a particular version of
Javelin was built, so that the distributor for a particular country could
build the version for that country.

Specifically referring to sorting, it became quite clear that we could not
depend on there being a canonical printable version of a sortable string.
In some languages, there are lower case characters without upper case
equivalents or vice versa.  The canonical form was a list of collating
sequence positions so that in English all versions of the letter "A" might
turn into 12, all versions of "B" into 13, and so on.  The canonical form
was easy to sort and useful for determining whether two strings were
equivalent, important in the symbol table, but you couldn't turn it back
into something printable.

Regards,
John Levine, johnl@iecc.cambridge.ma.us, {spdcc|ima|world}!iecc!johnl
Newsgroups: comp.std.internat
From: tt@tarzan.jyu.fi (Tapani Tarvainen)
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Originator: tt@tarzan.math.jyu.fi
Organization: University of Jyvaskyla
	<1992Dec31.203101.5447@prl.dec.com> <1i0s05INNnfn@rodan.UU.NET>
Date: Fri, 1 Jan 1993 11:56:37 GMT

In article <1i0s05INNnfn@rodan.UU.NET> avg@rodan.UU.NET (Vadim Antonov) writes:

>A good encoding should support easy (i'd say natural) localization.
>It should provide simple algorithms for simple functions
>like getting string length, searching a character, case-insensitive
>comparison, lexicographical comparison.

>Unicode (and for that matter Plan 9 UTF) does not support the last
>two mentioned functions. I have yet to see Plan 9 _sort_ which will
>sort Russian strings without being told explicitly that it is Russian.

So what?
I've yet to see anything even planned that would allow sorting
both Finnish and German without being told which is wanted.  
In fact I can't even imagine one that would make any sense.
In the case of a list of names, the very same data could be
sorted differently depending on where it is going to be used.

Tying sorting rules to character sets is not a good idea, IMHO.
--
Tapani Tarvainen  (tt@math.jyu.fi, tarvainen@finjyu.bitnet)
From: sommar@enea.se (Erland Sommarskog)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Date: 3 Jan 93 16:36:21 GMT
Organization: Enea Data AB

Dik T. Winter (dik@cwi.nl) writes:
>I said:
> > This is a simple end-user requirement which your proposal is not
> > incapabale to handle.
>I think you intended to say that Vadims proposal is not capable to
>handle it (I agree with that).

No. I did originally write "capabale", but then I realized I was wrong.
As I wrote it's perfectly possible to do it with a Vadim-set, but as
with Unicode, Latin-1 or any other set you have to tell the application
how to sort the data, so the benefits Vadim claims his ideas to have
are lost. (While the drawbacks remain.)
-- 
Erland Sommarskog - ENEA Data, Stockholm - sommar@enea.se
Jag gav en k{ck tjeck en check.
From: kiravuo@hut.fi (Timo Kiravuo)
Newsgroups: comp.std.internat
Subject: Re: Dumb Americans (was INTERNATIONALIZATION: JAPAN, FAR EAST)
Date: 1 Jan 93 15:47:05 GMT
	<8490@charon.cwi.nl> <1992Dec31.171450.1513@klaava.Helsinki.FI>
Reply-To: kiravuo@hila.hut.fi
Organization: Helsinki University of Technology, Computing Center

In article <1992Dec31.171450.1513@klaava.Helsinki.FI> wirzeniu@klaava.Helsinki.FI (Lars Wirzenius) writes:
>A-with-ring-on-top -- don't know its official name, if any.  The
>Finnish alphabet has no A-w-r-o-t, so I guess they have A-umlaut after
>Z, although they/we seem to usually include the A-w-r-o-t too, thereby

No, the Finnish alphabet is the same as the Swedish one. Letters
A to Z, A with ring, A with two dots, O with two dots. The
sorting is also done in this order, except V and W are usually
sorted (should be sorted) as one letter. If we use the currently
common 7-bit coding, characters ], [ and \ are replaced with A',
A" and O" (and }, { and | with a', a" and o", meaning that
sorting will not work with direct code sorting. Neither will it
in any different currently used coding (ISO 8859-1, DEC
Multinational, HP Roman-8, EBCDIC (the Scandinavian version we
are using), IBM PC or the Macintosh).

Also the telephone catalog seems to sort "Mc", "Mac", "van" and
"le" as they are written, but "von", "Von" and "der" according to
the main word.

Now if we also take into the account that French e' will usually
become E when capitalized and other well known charater behaviors
like the German double-s, it is my opinion that there is no way
to make a simple character code that will preform sorting and
character conversion automatically. 

So the ISO 10646 approach seems right to me. Make a character
code that codes just the appearance of the characters and leave
the rest for other people to worry about. My understanding is
that the making of ISO 10646 has been politically difficult
enough as it is, tryig to add more stuff into the standard would
just have blown the whole thing apart.

About the differences between Swedish and German (and Finnish)
A", I don't know why we should consider them as separate letters,
they are pronounced about the same way in all languages.  And
most of the other letters usually differ in other languages as
well. The Swedish letter O is pronounced like the Finnish U, in
English Y is as often concidered a vowel as not, while in Finnish
it is always a vowel etc.
--
Timo Kiravuo, kiravuo@hut.fi
Helsinki University of Technology, Computer Center, Finland
From: fvance@airgun.wg.waii.com (Frank Vance)
Newsgroups: comp.std.internat
Subject: ISO 8859-X What are the X's?
Keywords: ISO 8859
Date: 29 Dec 92 21:58:48 GMT
Organization: Western Geophysical, Div. of Western Atlas Int'l, Houston, TX

Can someone provide a brief overview of what the various parts of ISO
8859 are?  By this I mean, if 8859-1 is "Western European" and
-8 is "Hebrew" (as I have seen them called), what are the other parts?
-- 
Frank Vance  +1.713.963.2426		Western Geophysical
fvance@airgun.wg.waii.com		10001 Richmond Avenue
Fax: +1.713.963.2758			Houston, TX 77042   USA
From: andrew@alice.att.com (Andrew Hume)
Newsgroups: comp.std.internat
Subject: Re: An alternative I18N paradigm
Summary: yes, this is too naive.
Date: 28 Dec 92 06:45:54 GMT
Article-I.D.: alice.24495
Organization: AT&T Bell Laboratories, Murray Hill NJ



	no, you cannot simply wish the problem away by insisting
on a single locale for everyone. there simply exist different
local conventions and formats which may be politically, culturally, or
legally required. you (we) have to deal with them. the challenge is to
do so in a comprehensible fashion.

	andrew