Newsgroups: vmsnet.uucp
Subject: Re: NEWS / DECUS UUCP Integration
From: fenner@jazz.psu.edu (Bill Fenner)
Date: 13 Aug 92 15:57:51 EDT
Organization: Penn State University Mechanical Engineering

In article <1992Aug7.214814.618@cmkrnl.com> jeh@cmkrnl.com writes:
|This is an example of completely missing the point.  You said you ran NEWSSKIM
|in a separate queue.  I didn't know why, but I thought one reason might be to
|allow it to run even in the presence of hours' worth of RNEWS jobs.

No, the reason is to allow it to run at the same time as the incoming jobs.
If incoming jobs have to wait while the SKIM runs, we easily get behind.

Here's my setup:

NEWSQ - on ECLE, an 11/785.  NEWS_RNEWS runs here, as well as my DISTNEWS
ECLA_NEWSQ, ECLE_NEWSQ, ECLF_NEWSQ - all on 11/785's.  PROCNEWS runs in these
		queues.
POSTQ - on ECLA, UUCP outgoing and small outgoing NNTP feeds run in this queue
POST_HRB_Q - on ECLA as well, our big NNTP feed runs in this queue.

DISTNEWS uses two logicals to decide how to divide up the work - one list of
nodes, one list of weights.  Right now, the list of nodes is
ECLE,ECLF and the weights is 2,2.  When we get behind, we up the list to
ECLA,ECLE,ECLF and the weights to 1,2,2.  [1 because ECLA has lots of other
stuff to do.]  If we get really far behind, we could add one of the 8550's
in the cluster in, but that's a last resort.  DISTNEWS then renames NNTP
incoming files to include the node name of the ndoe that should process
them, checks for sufficient disk space, and submits a PROCNEWS job to
'node''_NEWSQ to process the news.

When the PROCNEWS job finishes, it submits another DISTNEWS (DISTNEWS uses
locking to be sure that it only runs after all PROCNEWS jobs finish), which
processes the log files for the most recent adds for errors, submits a job
to POSTQ and POST_HRB_Q to send out the news that was just processed, then
goes to work distributing news again.

I modifed NEWS_RNEWS to just rename the UUCP inbound job to look like an
NNTP batch file so DISTNEWS will pick it up.


The point of all this explanation is so that you will understand what my
situation is.  We've been wasting both of our time while I've been giving
incomplete examples of my setup and you've been guessing at the rest.

Here's an example of what's running on a mostly-typical, just-barely-behind
day:

Queue           Username     Job Name             Entry Status
--------------- ------------ -------------------- ----- ------------
ECLE_NEWSQ      NEWS         News processing        725 Executing
ECLF_NEWSQ      NEWS         News processing        724 Executing
NEWSQ           NEWS         CHKTIME                708 After 11-AUG 15:21
POSTQ           NEWS         NEWS post              409 Executing
POSTQ           NEWS         NEWS post              496 Pending (queue busy)
POSTQ           NEWS         NEWS post              603 Pending (queue busy)
POSTQ           NEWS         NEWS post              719 Pending (queue busy)
POST_HRB_Q      NEWS         NEWS post to HRB       497 Executing
POST_HRB_Q      NEWS         NEWS post to HRB       604 Pending (queue busy)
POST_HRB_Q      NEWS         NEWS post to HRB       720 Pending (queue busy)
SKIMQ           NEWS         NEWS skim              741 After 11-AUG 23:59

|>Well, NEWS_ROOT and the dreaded indexed files are also on this disk, so making
|>it a bound volume set would probably hurt.
["This disk" is "the nearly empty RA81 with NEWS_MANAGER on it"]
|
|You have also missed another key point:  From a disk fragmentation standpoint
|it is nearly essential to keep the indexed files on a separate volume from the
|item files.

They are.  The indexed files are on DSKM:.  NEWS_MANAGER is on DSKM:.  The
items are on DSKN:.

I'm moving news_manager to dskn:, we'll see how it goes.

  Bill
--
Bill Fenner	fenner@jazz.psu.edu	..psuvax1!hogbbs!wcfpc!wcf
		wcf@hogbbs.scol.pa.us   (+1 814 238-9633 v.32bis)