HDTenEightyP/NetNews huggingface dataset:
From: kosta@blues.kk.sub.org (Kosta Kostis)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Re: Data tagging (was: 8-bit representation, plus an X problem)
Keywords: magic codes, portable data
Date: 20 Dec 92 01:52:45 GMT
Organization: The Blues Family

andersa@Riga.DoCS.UU.SE (Anders Andersson) writes:

> [note Followup-To: comp.std.internat]
> 
> In article <1gt5a2EINNin3@uni-erlangen.de>, unrza3@cd4680fs.rrze.uni-erlangen
> > It should also be noted, that at least one existing OS (Windows NT)
> > uses a 2 byte encoding both internally (e.g. in filenames in Fnodes
> > on the disc) as well as in text files. Text files always begin with
>                           ^^
> > FEFF as a magic code for ISO 10646 textes. This code also indicates,
> > whether it is a littleendian file.
> 
> Is this magic code visible to the user without any special tricks,
> or is it filtered away by the operating system when the file is
> opened for reading?  Suppose I obtain a file, that is labeled as
> containing IS 10646 text, via FTP from a server running Windows NT,
> to a client running a different system--will I then get this 0xFEFF
> magic code (which is meaningless on my system) too, or will I get a
> 'clean' IS 10646 text?

Well, I guess programs will "see" either "0xFEFF" or "0xFFFE" depending
on whether the file has been written by a big or a little endian.
If you have a system for which ISO 10646 text files is not meaningless,
the "magic" code isn't meaningless.

> I remember seeing text files containing an explicit ^Z (0x1A) at
> the end, due to their origin on some home computer where ^Z was the
> ordinary EOF marker, even though I was sitting on a system with
> perfectly functional EOF pointers in the file descriptor blocks...

Never mind. That's a totaly different story. 

> I hope the above isn't yet another version of that problem (non-
> standard tags or markers floating around with standards-compliant
> data on systems not understanding them)?

The tag *is* defined the UniCode standard and so it should be defined
in ISO 10646, too (this is an assumption). All systems capable of
reading/displaying UniCode/ISO 10646-16-bit (the latter is not the
official name) should follow this tagging.

> Alternatively, does this magic code have any chance of becoming
> a standard itself?

See above.

        Kosta


-- 
  Kosta Kostis, Talstrasse 25, D-6074 Roedermark 3, Germany
  kosta@blues.kk.sub.org                                        (home)
  sw authors: please support ISO 8859-1!        dv|DV\_=aeoeueAEOEUEss
From: erik@poel.juice.or.jp (Erik M. van der Poel)
Newsgroups: comp.std.internat
Subject: Re: 8-bit anything?
Date: 12 Dec 92 03:58:29 GMT
Organization: sometimes

> if a message was tagged (say as being latin1, or shift-JIS, or big5 or
> 8859-9), we have no problem

The problem is that, currently, not very many messages are tagged.
Hopefully, MIME tagging will become more widely used soon.  But there
are other ways of determining the character sets used in messages.
For example, the Japanese have been using an encoding now often
referred to as "ISO-2022-JP", that contains escape sequences that
indicate the character set of the following bytes.


> question 1 is: is this a plausible tactic, and is there a standardised
> list of character sets that are likely to be supported (or even
> better, must be supported)?

That depends on what you actually want to do.  For example, if one of
the researchers at your lab can read Japanese and wants to subscribe
to a mailing list in Japan, then that person should have ISO-2022-JP
software installed.  On the other hand, if someone wants to read the
Japanese newsgroups (such as fj.*), then you need to arrange a
newsfeed from some site that carries fj.* and whose upstream sites
don't strip ESCAPE.


> question 2: given a plausible answer to question 1 (say, MIME), would
> someone with such a system volunteer to conduct some tests of it with
> me?

I can send you ISO-2022-JP mail, with or without MIME headers.  I can
introduce you to some Japanese mailing lists.  I don't actually know
how to get fj.* to you, but I'm sure I can find out, if needed.


> question 3: how do i go about registering a sensible encoding (like
> the FSS-UTF mentioned above) so others can use it?

MIME charsets are registered using the procedure described in MIME
itself.  You may want to circulate a draft before registering the
document.
-
-- 
Erik M. van der Poel                               erik@poel.juice.or.jp
Xref: sparky comp.std.internat:882 news.admin.misc:805
From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Re: 8-bit news
Keywords: ISO8859 ISO10646 8bit
Date: 17 Dec 92 01:32:14 GMT
Organization: Silicon Graphics, Inc.  Mountain View, CA

In article <89RsVB3w165w@blues.kk.sub.org>, kosta@blues.kk.sub.org (Kosta Kostis) writes:
> ...
> If a system is not capable of storing and forwarding 8-bit octets,
> why is it connected to the Internet? Maybe we should simply not route
> anything over those systems. Old hardware got to be replaced someday
> and should it be only because of the energy it consumes...  ;-)
> ...

This is a persistent cause of the continuing disagreement.

Some modest corners of the Internet are "authoritarian."
Someone or some group tells everyone what they can do.

Other parts are more "democratic" or even "anarchic".  No one can tell
anyone else what to run in those corners.  At most, "peer pressure" can
be invoked against those who do not run "standards compliant" systems.

For now, "standards compliant" means "7-bit."  You might disagree with
the wisdom of either of those facts (that "the net" is not run by a
benovolent European dictator or that 7-bits is or ever was a reasonable
standard).  It is crazy (as is in "refusing to accept reality") to say
that either is not a fact.

SMTP was developed by people were not crazy, who know that some target
machines could at the time only do 7-bit text, and only with some
difficulty.  MIME is being developed by people who are similarly
pragmatic.  MIME will move 8-bit mail through systems that are both
old and 7-bit.

It was probably either provicial or short sighted for SMTP to not have
included some kind of encoding for 8-bit data (although the "S" does
stand for "simple").  It is provincial today to expect the entire
Internet to be changed to support 8-bit text; It is provincial because
it ignores the concerns and constraints of other people.


Vernon Schryver, vjs@sgi.com
Xref: sparky comp.std.internat:831 news.admin.misc:725
From: leonard@qiclab.scn.rain.com (Leonard Erickson)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Re: 8-bit representation, plus an X problem
Date: 14 Dec 92 06:08:18 GMT
Article-I.D.: qiclab.1992Dec14.060818.13750
Reply-To: Leonard.Erickson@f51.n105.z1.fidonet.org
Organization: SCN Research/Qic Laboratories of Tigard, Oregon.

kosta@blues.kk.sub.org (Kosta Kostis) writes:

>Right. Who maintains the fidonet/usenet gateway software?
>I'd love to send him/her two nice conversion tables from/to
>ISO 8859-1.

Too many people. That's like asking "who maintains the news software?"

A quick grep through the latest nodelist shows *80* official gateways.
-- 
Leonard Erickson		      leonard@qiclab.scn.rain.com
CIS: [70465,203]			 70465.203@compuserve.com
FIDO:   1:105/51	 Leonard.Erickson@f51.n105.z1.fidonet.org
(The CIS & Fido addresses are preferred)
Xref: sparky comp.std.internat:912 news.admin.misc:850
From: andersa@Riga.DoCS.UU.SE (Anders Andersson)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Data tagging (was: 8-bit representation, plus an X problem)
Followup-To: comp.std.internat
Date: 19 Dec 1992 00:53:01 GMT
Organization: Uppsala University, Sweden
Distribution: world
NNTP-Posting-Host: riga.docs.uu.se
Summary: In-band or out-of-band, that's the question...
Keywords: magic codes, portable data

[note Followup-To: comp.std.internat]

In article <1gt5a2EINNin3@uni-erlangen.de>, unrza3@cd4680fs.rrze.uni-erlangen.de (Markus Kuhn) writes:
> It should also be noted, that at least one existing OS (Windows NT)
> uses a 2 byte encoding both internally (e.g. in filenames in Fnodes
> on the disc) as well as in text files. Text files always begin with
                          ^^
> FEFF as a magic code for ISO 10646 textes. This code also indicates,
> whether it is a littleendian file.

Is this magic code visible to the user without any special tricks,
or is it filtered away by the operating system when the file is
opened for reading?  Suppose I obtain a file, that is labeled as
containing IS 10646 text, via FTP from a server running Windows NT,
to a client running a different system--will I then get this 0xFEFF
magic code (which is meaningless on my system) too, or will I get a
'clean' IS 10646 text?

I remember seeing text files containing an explicit ^Z (0x1A) at
the end, due to their origin on some home computer where ^Z was the
ordinary EOF marker, even though I was sitting on a system with
perfectly functional EOF pointers in the file descriptor blocks...

I hope the above isn't yet another version of that problem (non-
standard tags or markers floating around with standards-compliant
data on systems not understanding them)?

Alternatively, does this magic code have any chance of becoming
a standard itself?
--
Anders Andersson, Dept. of Computer Systems, Uppsala University
Paper Mail: Box 325, S-751 05 UPPSALA, Sweden
Phone: +46 18 183170   EMail: andersa@DoCS.UU.SE
Newsgroups: comp.std.internat
Subject: Re: ANSI/MIL-STD-1815A (Ref Manual for Ada PL)
Organization: I.E.C.C.
Date: 15 Dec 92 17:19:31 EST (Tue)
From: johnl@iecc.cambridge.ma.us (John R. Levine)

In article <936@gofish.cards.COM> you write:
>Does anyone know where I can get a copy of ANSI/MIL-STD-1815A, a.k.a.
>Reference Manual for the Ada Programming Language?  Hard-copy, soft-copy...
>any kind of copy (tho' an ftp site is preferred).  

If it's an ANSI standard, you order it from ANSI.  Their number is
212-642-4900.  My catalog says it costs $25 plus a few bucks for postage
and handling.

ANSI derives most of its income from the sales of its copyrighted
standards, so they're not available electronically for free.

Regards,
John Levine, johnl@iecc.cambridge.ma.us, {spdcc|ima|world}!iecc!johnl
Xref: sparky comp.std.internat:842 news.admin.misc:742
From: peter@ferranti.com (peter da silva)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Re: 8-bit news
Date: 14 Dec 92 23:02:25 GMT
Organization: Xenix Support, FICC

In article <1992Dec13.100023.7056@clarinet.com> brad@clarinet.com (Brad Templeton) writes:
> We're not running the same sort of net any more.  So many links are tcp/ip
> now.

That's the problem. UUCP is 8-bit clean. It's TCP/IP that's most likely
to trash articles.
-- 
%Peter da Silva/77487-5012 USA/+1 713 274 5180/Have you hugged your wolf today?
/L{lineto}def/C{curveto}def/F{0 562 moveto 180 576 324 648 396 736 C 432 736 L
482 670 518 634 612 612 C}def/G{setgray}def .75 G F 612 792 L 0 792 L fill 1 G
324 720 24 0 360 arc fill 0 G 3 setlinewidth F stroke showpage % "Peerless"
Xref: sparky comp.std.internat:872 news.admin.misc:787
From: keld@login.dkuug.dk (Keld J|rn Simonsen)
Subject: Re: Week Numbering
Organization: DKnet
Date: Fri, 18 Dec 1992 16:15:37 GMT

tml@tik.vtt.fi (Tor Lillqvist) writes:

>In article <1992Dec18.105632.2494@odin.diku.dk> torbenm@diku.dk (Torben AEgidius Mogensen) writes:
>   >Is there an international standard for numbering weeks.

Yes, the standard is ISO 8601.

>It is interesting to note that ANSI C and POSIX strftime(3) uses the
>wrong week numbering algorithm, at least according to its manual page
>in HP-UX, which says:

There will be new formatters in POSIX to comply with ISO 8601.
The old behaviour is reatined with existing formatters.

Keld Simonsen
From: torbenm@diku.dk (Torben AEgidius Mogensen)
Newsgroups: comp.std.internat
Subject: Re: Week Numbering
Date: 18 Dec 92 10:56:32 GMT
Organization: Department of Computer Science, U of Copenhagen

andrew@alarbus.oz.au (Andrew Reye) writes:

>Is there an international standard for numbering weeks.

>Some 1993 diaries have the week with Mon 4th Jan as week
>1 and otheres have it as week 2.

>Is this different in different countries ???

I don't know if it is different in different countries, but at least
in Denmark the rule is that the week with the first Thursday is week
1. If the 1st of January is Friday through Sunday, this is in week 52
or 53. In 1993 it is week 53, in 1994 it will be week 52.

Some printers of calenders in Denmark made the mistake of printing
large quantities of calenders with Jan. 1st being in week one. They
had to recall these. and sell them with warning labels at much reduced
prices. Such an error may have caused the discrepancy between the
diaries you have seen.

	Torben Mogensen (torbenm@diku.dk)
Xref: sparky comp.std.internat:820 news.admin.misc:710
From: erik@poel.juice.or.jp (Erik M. van der Poel)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Re: 8-bit news
Date: 12 Dec 92 12:42:04 GMT
Organization: sometimes

> What character set(s) do those Japanese language newsgroups use?

They use Kanji, Katakana, Hiragana, and other characters found in JIS
X 0208, JIS X 0201 and ASCII.  Romaji is rare, except in message
headers. :-)

The characters take up either one or two bytes each.  The 8th bit is
not used.  Escape sequences are used to delimit parts in different
character sets.
-
-- 
Erik M. van der Poel                               erik@poel.juice.or.jp
Newsgroups: comp.std.internat,news.admin.misc
From: atkinson@itd.nrl.navy.mil (Randall Atkinson)
Subject: Re: 8-bit news
Keywords: ISO8859 ISO10646 8bit
Organization: Naval Research Laboratory, DC
Date: Tue, 15 Dec 1992 20:56:34 GMT

In article <89RsVB3w165w@blues.kk.sub.org> kosta@blues.kk.sub.org (Kosta Kostis) writes:

>>   I *KNOW* of at least 3 systems (combinations of OS + hardware) that
>> will flat out crash if offered certain fairly common 8-bit
>> ISO-8859-like characters.  I'm not going to say which ones in public
>> because some lurker would go and use this to crash someone else's
>> machine maliciously.

>I have never said one can do that *without* changing some software
>at some places. So, what's the "nonsense" in here?

  The systems are mostly orphans.  In some cases, the vendor doesn't
even exist anymore.  The software effectively cannot be changed.  Such
systems are a large part of the reason that the DDN has had an awful
time trying to convert to the Domain Name System.  Heck some of the
older systems from still existing vendors used 6-bit characters rather
than 7-bit ASCII.  Ever programmed on a Cyber ?  Mayhaps not.  However
many such systems won't go away or be replaced soon for financial and
other reasons.

>If a system is not capable of storing and forwarding 8-bit octets,
>why is it connected to the Internet? Maybe we should simply not route
>anything over those systems. Old hardware got to be replaced someday
>and should it be only because of the energy it consumes...  ;-)

  The Internet does not require that any machine use any particular
internal word size or character size.  No RFC requires the use of
8-bit characters and no standards-track RFC is likely to require that
soon, if ever.

  Because of dynamic routing, the vagaries of MX handling, and other
factors, it isn't really possibly to prevent mail traffic from being
misrouted via any particular host.  This is why the ESMTP draft on
8-bit extensions REQUIRES that the sending host do a handshake to
verify that the receiving host says that it can handle 8-bit BEFORE
any 8-bit gets sent.  News uses a flooding algorithm for article
propogation and so there isn't really much user control over routing.

>>   The second is to have a standard for marking the contents of
>> messages in standard headers using standard syntax and standard
>> tokens.  If I understand Henry correctly, he is putting together a
>> draft, based on MIME, that addresses this second issue.

>Didn't MIME specify some encodings already? And wasn't ISO 8859-1
>the default for 8bit encoded text? (hm...)

MIME only talked about Internet email, not news.  Henry's draft will
apparently talk about news issues and so forth.


Xref: sparky comp.std.internat:826 news.admin.misc:720
From: kosta@blues.kk.sub.org (Kosta Kostis)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Re: 8-bit representation, plus an X problem
Keywords: ISO8859-1 CP850 fidonet gateway
Date: Sun, 13 Dec 92 13:10:57 MET
Organization: The Blues Family

ppan@nobeltech.se (Per Andersson) writes:

> In article <kkZNVB5w165w@blues.kk.sub.org> kosta@blues.kk.sub.org (Kosta Kost
> >Yuk! Allthough I'm sitting in front of an IBM PC clone using
> >IBM Code Page 850, I would never think of bothering the rest 
> >of the world with characters between 128 and 159. That's why
> >I convert between CP850 and ISO 8859-1.
> 
> At least you know that you are using codepage 850. Lots of people are
> not aware that the IBM PC uses several codepages, and fidonet has to my
> knowledge no handling of the different codepages, not to mention other
> charactersets as Latin x ( I think Latin-1 exists as CP819 or something),
> Unicode 1.1 or ISO10646.

All displayable characters in ISO 8859-1 can be converted to IBM Code
Page 850 and then back to ISO 8859-1 without any problem. The characters
are arranged in a different order to maintain backward compatibility with
IBM Code Page 437 (US). In Code Page 850 mainly some graphics characters
have been replaced by Latin 1 characters.

I used to use IBM Code Page 437 for years, because I never had a reason
to switch to IBM Code Page 850. As soon as I discovered IBM Code Page 850
has all the characters found in ISO 8859-1 (Latin 1), I instantly made
the switch. The installation procedure for MS-DOS 5.0 does install
IBM Code Page 850 (in Germany) anyway.  ;-)

>                          Also many people in fidonet, at least in sweden 
> are not interested in interoperability, and could care less about the 
> millions of users reachable on the Internet. Hopefully not all fidonet
> users have this attitude.

I am looking for the authors of fidonet/usenet gateway software to give
them a conversion table so fido articles do generate ISO 8859-1 characters.
I already asked in comp.bbs.waffle (thanks for the hint) - is there any
other suitable place to ask?

After that, we have to find the maintainers of fidonet/usenet gateways. :-)

        Kosta


-- 
  Kosta Kostis, Talstrasse 25, D-6074 Roedermark 3, Germany
  kosta@blues.kk.sub.org                                        (home)
  kostis@frsmic.enet.dec.com                                    (work)
  sw authors: please support ISO 8859-1!        =aeoeueAEOEUEss
Xref: sparky comp.std.internat:886 news.admin.misc:811
From: andrew@alice.att.com (Andrew Hume)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Re: 8-bit representation, plus an X problem
Summary: clarification
Keywords: ISO8859-1 CP850 fidonet gateway
Date: 17 Dec 92 05:36:10 GMT
Article-I.D.: alice.24426
Organization: AT&T Bell Laboratories, Murray Hill NJ

In article <1gnemoEINN3qo@uni-erlangen.de>, unrza3@cd4680fs.rrze.uni-erlangen.de (Markus Kuhn) writes:
> ...
> in the above pipeline. I heard roumors, that the Plan 9 UTF version will
> be added as UTF-2 to an ISO 10646 annex, can anyone confirm this? Perhaps

	firstly, 10646 is cast in concrete right now. there will
be no changes for at least a year or two or three.

	secondly, the UTF encoding is now formally called UTF-1.

	thirdly, the Plan 9 encoding is the same as the FSS-UTF
encoding being considered for adoption by X/Open as the
recommended encoding for 10646 text streams. (i believe it
has been approved by the technical committee and is wending
its way through some process inside X/Open.)

	fourthly, if as we expect X/Open adopts FSS-UTF, it will
be proposed as an addition to 10646 as an alternate UTF encoding
(presumably called UTF-2 -- we have heard of no other encodings).
This could be done relatively quickly and expeditiously if ISO
so desired; otherwise, it will wait until the next time someone tweaks
10646.

> we really should wait as suggested by Randall Atkinson for the final
> version of ISO 10646 before starting to include anything new (e.g.
> a 16-bit encoding) in MIME.

	this won't help. the character assignments are known; translation
tools exist (i have nearly got permission for public release of the
Plan 9 translation stuff); and the ISO-blessed encoding (UTF-1) is known.

> Of course. But USENET users will have difficulties in understanding this,
> as there isn't currently anything like a presentation layer (as defined in the
> OSI reference model) that performs the conversion between the local
> representation (e.g. I prefer Latin 1 files) and the encoding used on
> the network (e.g. any 8-bit encoding on news and 7-bit encoding on
> email). ...

	maybe some usenet users will have difficulties. i suspect most
won't. but in any case, they oughtn't have to know! mail has
always struct me as one of the easiest programs to control in this way
as typically all mail leaves and enters a system through a couple
of programs. and conceptually, those programs could easily and
transparently handle conversion to and from a transport format/encoding.

	right now, we have the transport model/encoding and conversion
stuff; all we lack are the conventions to get 8-bits thru cleanly
and to do the translations.

			andrew
Xref: sparky comp.std.internat:818 news.admin.misc:707
From: jbuck@forney.berkeley.edu (Joe Buck)
Newsgroups: comp.std.internat,news.admin.misc
Subject: Re: 8-bit news
Date: 12 Dec 1992 20:40:00 GMT
Organization: U. C. Berkeley
NNTP-Posting-Host: forney.berkeley.edu

Given: we want to update to 8-bit news.

Also given: quite a few sites will not update, no matter how much we
scream.  How to solve this?  Here's a suggestion:

For any news article that has 8-bit characters in it, include a special
header indicating the fact.  This header itself will contain eight-bit
characters, and will be structured in such a way that it is very likely
to be trashed by a 7-bit link, but that a recognizable residue will
come through.  Something like:

Character-Set: <stuff with 8-bit characters and a checksum here>

This header can also be used to specify which character set is in use,
where it matters.

Now there are three cases:

First, the article may go through an 8-bit-clean link.  If so, it comes
out OK.

Second, the article may go through a 7-bit link with intelligent
system administrators on both ends.  The articles are encoded using
some protocol and decoded on the other end, so that an 8-bit-clean
link is produced on top of a 7-bit channel.

In either of these cases, the article comes out unscathed.  The final case
is the one everyone is worried about: such articles go through a seven-bit
link using old software.  At the other end, the Character-Set header is
altered: the leading "Character-Set:" comes through, but what follows will
not be correct.  Sites with up-to-date software know that such articles
are likely to be corrupt, and simply drop them on the floor.
--
Joe Buck	jbuck@ohm.berkeley.edu
Xref: sparky comp.std.internat:812 news.admin.misc:697 comp.protocols.iso.x400:704