From: mark@infocomm.com
Newsgroups: news.software.anu-news
Subject: Re: Articles overflow directory limit
Date: 15 Oct 92 00:23:10 PDT
Organization: INFO COMM - Computer Consulting, Redwood City, Ca

In article <Bw2sx7.L13@cs.psu.edu>, fenner@postscript.cs.psu.edu (Bill Fenner) writes:
> In article <1992Oct13.102130.210@indyvax.iupui.edu> imhw400@indyvax.iupui.edu writes:
> |o	Keep the existing directory structure to keep articles in various
> |	groups segregated.  This is a good fit to the organization of
> |	the groups, and if you lose one file it's no big deal to restore it.
> |
> |o	Each group's text is stored in a VMS text library:
> |	NEWS_DEVICE:[group...group]ITEMS.TLB.
> 
> Well, it's kind of silly to have a directory with one file in it.  How about
> NEWS_DEVICE:[all.but.last]COMPONENT.TLB, so, for example, the directory
> NEWS_DEVICE:[news.software] would have READERS.TLB, B.TLB, ANU-NEWS.TLB, etc.

Um, err, well . . .  Before you go off into refining details, there is at least
one giant hole in the general feasability of using "VMS text libraries" to 
store article data.  Has anyone ever tried to install a VMS layered product
(say VAX C) while someone on the system using the Help command?  The Help
command uses a form of "text libraries", help libraries, and the layered
product installation will fail due to the inability of installation procedure
to update or initially install the product's DCL help.  The problem is that the
library is really a good means of reading data, but not so good at updating and
revising.  The problem is that NEWS tends to arrive constantly and hence there
would be a constant need to update and revise the contents of the text
libraries and this would be impossible if anyone one happened to be running 
news.

The general problem of file system overhead being a bottleneck to news
processing efficency is undeniable.  Jamie Hanrahan proposed (several years
ago) an idea for an alternate storage methodology.  The idea is built around
the concept that articles arrive in "batches", why not leave them that way. 
The current index file structure might not have to change at all or certainly
very much, while the types of things that would need to change would include:
1) how an articles contents are located (which batch file, and how far into it)
2) how batch file maintenance is performed (a new approach to FILE and
DIRECTORY and ITEM skimming), and 3) some concept of reference counts for
batches (how many items they contain, how many pointers there are to them,
etc.), sparcely referenced batches might have their unreferenced data
"squeezed" out during maintenance, small batches might be merged, etc.
In any case, reorganizing the storage methodology is a non trivial task which
will eventually be necessary given the fact that traffic volume is growing 
faster than disk access speeds are getting faster.  The volume will eventually
catch up.  We are probably a reasonable number of years from this situation
being critical, and it will really only be an issue for sites that want/need to
carry ALL the traffic.

The problems of how NEWS traffic stresses a file system capabilities is by no
means unique to ANU-NEWS.  Unix news system administrators have similar
problems with "running out of inodes", etc.  The real issue is that the 
nature of NEWS as an application, and how it references disk storage is
probably distinctly different than your more traditional every day use of your
computing environment.  This context of use currently requires some planning
and/or special maintenance measures.

--
Mark Pizzolato - INFO COMM Computer Consulting, Redwood City, Ca
PHONE:	(415)369-9366	UUCP:  decwrl!infopiz!mark or uunet!lupine!infopiz!mark
DOMAIN:	mark@infocomm.com