Newsgroup: it.news.net-abuse
Titolo: [forward] Usenet Spam Cancel Moratorium


Subject: Usenet Spam Cancel Moratorium - Where next?
[CC'd to Usenet, and various relevant mailing lists.]
The Spam Cancel Moratorium on Usenet been on almost two weeks, and
it's past time for some serious analysis discussion of what happened,
what it means, and what we do with what we've learned.
We had originally anticipated that the spam cancel moratorium would
be in effect for two weeks (ending tomorrow (er, today ;-)), however,
events as they occured may make the idea of "ending" it moot.
The idea behind the Moratorium is simple: 200,000 cancels per day
and climbing is getting absurd and very damaging all by itself.
Too many ISPs (and users for that matter) are relying on spam
cancellation (and the despammers themselves) to keep their discussion
areas viable.  It has always been my (and I believe all other
despammers) intent to put ourselves out of business.  Spam cancellation
was supposed to be a temporary measure until ISPs and other sites got
their act together and maintained the viability of Usenet on their
own.
This wasn't happening in the status quo prior to the moratorium.  The
situation wasn't improving, worse, the "temporary solution" is
becoming an enormous problem all of its own.
Hence, something had to be done to shake up all of Usenet, and getting
systems to start seriously looking at ways to solve things.
Because sooner or later, all of the despammers are going to give up
permanently.  The cost of despamming (both in required resources and
personal commitment) is climbing fast to ridiculous heights.  There is
also considerable frustration because there still is no end in sight.
So, on April 3, most of us simply stopped issuing spam cancels.
We still generated NoCeM notices (which are not a particularly
good solution either, but one of the best currently available),
and continued complaining about spam - as everyone should who
cares about the continued viability of Usenet.  It was time
to try to force some evolution.
I hope that this posting can generate some serious discussion
about where we go from here.
I should point out that this posting is my own observations and
conclusions, rather than a "group statement".  However, I think it's
fair to say that most of the despammers who participated in the
discussions leading to the moratorium agree with much of this.
What happened:
        1) Spam volumes jumped substantially.  Systems without filtering
           of kind have reported traffic volumes going up by 20% or
           even as high as 400%.
        2) As one datapoint, the detection rate on my spam cancelling
           system went up by a factor of 20 and has stayed there for
           two weeks.
           Similarly, systems with good filtering (such as SpamHippo
           or CleanFeed) have reported substantial jumps in local
           reject rates.
           This demonstrates very clearly the effectiveness of the
           pre-Moratorium despammers of cancelling articles before they
           arrived on many systems.
        3) A number of systems overflowed, including one of mine, and
           one of Joe Greco's.  There are all sorts of rumours flying
           around that "Australia is down", "Usenet backbones in Europe
           have crashed", " there are loading problems at UUNET", and
           we've seen many reliable reports of individual systems
           grinding to a halt (including major ones such as one of
           MCI's even after implementing a 6 hour expiration on alt
           groups).
           Going by the fairly reliable statistic of Usenet
           consisting of 40% spam, 40% spam cancels, and 20% content,
           we immediately see that an entirely unfiltered system
           would be forced to store three times as many articles if
           the cancels weren't sent.  In other words, tripling
           the storage requirement of a news server.
           It's not quite that simple, of course, because the cancels
           aren't occupying space, but as a significant portion of
           spam is vastly larger than the cancels, and most systems
           are expiring cancels _very_ quickly, the effect is fairly
           negligible.
           Factoring in Cosmo's 24 hour cancel delay, we see that
           the "theoretical unfiltered server" would be required to
           store triple the normal daily volume of articles received
           for one day, and approximately double the normal daily volume
           of articles (using a ballpark figure of Cosmo getting 50%
           of spam) for each additional day of retention the server
           has.
           It's rather difficult to tell whether any one of these system
           failures are directly a result of the Moratorium.  Few system
           administrators will publicly announce that their servers have
           failed to meet the test.  But some did so.
           However, with increased storage requirements such as this,
           it is bound to have been a factor.  It certainly was on
           ours.  Over 500Mb in alt.mag.playboy in one day, where
           something like 75-100Mb is more normal pre-moratorium.  We
           only get 6,000 groups and we have a 13Gb spool, but, I still
           have to expire alt in one day now.  This is only on the
           anti-spam server.  Our reader servers have filtering, and
           didn't seem to see any big effects.
        4) Many systems report having to drastically decrease expiration
           times.  In some cases retention of articles were shortened to
           day or even less (MCI to 6 hours in alt) to cope with
           increased storage requirements.  Other systems had to
           increase the frequency of expiration invocations.  One had
           to resort to hourly emergency expirations.
        5) We're hearing many reports of systems implementing filters,
           including major systems such as Digex, and mid-tier
           ISPs taking feeds from MCI.  There are also reports of jumps in
           the retrieval of spam filtering software from their various
           web repositories.  This was one of the main goals of the
           Moratorium, and is most heartening.
           [Coincidentally, AOL's filtering system made it's long
           awaited debut.  While they're not removing spam from
           their systems, similarly to Dejanews, they're at least
           providing their users with a choice to see spam or not.
           Their spam detection is at least in part via NoCeM.
           I say "coincidentally", because this had been in the
           works for quite some time, and is clearly not a result
           of the moratorium.  But the timing was most opportune ;-)]
        6) Visible spam differences....
           We knew ahead of time that the vast majority of spam (around
           90-95%) appears in alt.* sex-related groups.  These are the
           very groups that most people don't read, or those that do,
           won't admit to it publicly ;-)  Furthermore, many systems
           (perhaps even a majority) don't receive these groups at all,
           so their users won't see much difference.
           So, there weren't huge numbers of users complaining about
           spam volumes.  There were many.  But, a somewhat disappointing
           turnout.
           Furthermore, many users are on systems that are already
           filtering (such as Netcom, Zippo/Newsguy, Mindspring etc.),
           without even knowing it.  These systems are already acting
           responsibly, and were never the target of the moratorium.
           One prime example of a group where the difference is
           obvious: alt.sexual.abuse.recovery.  Prior to the
           moratorium, at most times, you'd be hard pressed to find
           more than one or two spam articles there per day.  During
           the moratorium, a typical day would have 100 postings, 95 of
           which were spams (usually pornographic pictures).
        7) Cosmo.  Cosmo Roadkill is presently responsible for
           approximately 50% of all Usenet spam cancels.  Much has been
           made about the fact that Cosmo decided to not participate in
           the moratorium the same way as the rest of us.  What he did
           do was to delay the propagation of his cancels for 24 hours.
           Hence, all spam would survive on a server for 24 hours, and
           approximately 50% would disappear afterwards.
           This still served the purpose of the Moratorium.  We're
           certain that the effects would have been a lot more
           pronounced if he had simply ceased cancelling altogether.
           However, the effects are incontrovertible anyways.
           We had considered the possibility of Cosmo not particpating
           at all prior to the April 3 Moratorium commencement, because
           he was not responding to our queries.  The concensus was to
           continue regardless.
           The most significant effect of Cosmo's partial
           participation was to confuse people (especially the media -
           some media really are confused.  Including the one saying
           "The Moratorium Failed!", citing as proof Demon didn't see
           an increase in _email_ spam.   Sigh...  At least Brock Meeks
           has reversed his position on the Smith Bill, so at least
           there are at least some signs of intelligence out in
           medialand ;-)
           [To be fair, most reporting of the Moratorium was reasonable.]
        8) Our fellow despammers: many major despammers have already stated
           their intention to not resume issuing cancels, at least
           temporarily.
What We've Learned / What Next:
    1) More systems than we thought were already filtering, or were
       otherwise less affected by spam than anticipated.  Still,
       many major systems had serious problems - we hope they've
       learned something from this and are looking more seriously
       at filtering.
    2) Systems _do_ take notice of these things.  The increased
       deployment of, and interest in, filtering is clear.
    3) This has heightened interest in evolving Usenet itself,
       to things like the Usenet 2 project.  This sort of thing
       is slow to grow, issuing cancels and propping up Usenet 1
       simply discourages it from growing fast.
    4) People don't complain much about deluges of spam in alt,
       except in some limited areas.  I believe that this proves
       several things:
            a) Most of alt is hopelessly doomed and is overrun
               by spam.  We've lost that battle, and it's probably
               no longer worth trying to prop up.  This idea
               was first posited a few years ago.  I think we have
               the proof now.
            b) Our efforts in alt would probably be better spent trying
               to rescue salvagable groups (such as
               alt.sexual.abuse.recovery) and put them on firmer
               footing - either by migration to some other hierarchy
               (at the same time removing the three consecutive letters
               "s", "e" and "x" from it's name), or by moderation.
            c) The existing alt hierarchy will simply die off as more
               and more sites decide that they can no longer afford to
               propagate it.  Whole countries have already done this.
               Issuing 100s of thousands of cancels per day for spams
               in alt are simply prolonging the agony.
       Alt is dead, it's time to quit flogging the corpse.
       There are still many viable pockets of discourse in alt.  These
       groups should start considering when whole networks of systems
       start dropping alt in-toto, and thinking about moving to a
       better neighborhood.
    4) There really isn't a lot of spam in non-alt groups (except in
       some regionals and misc.jobs.*).  This isn't a surprise -
       we already knew this.
       Many (or perhaps most) regionals are already better monitored
       by local administrators than we can possibly do.  Many have
       implemented considerably stricter rules than we apply.
       They need relatively little help from us.
       The volume of spam in the mainstream global groups (such as
       the big 8) is perhaps small enough to not currently warrant
       active spam cancellation.
Some personal (possibly widely shared) conclusions:
We went into the moratorium knowing that there were four
possible outcomes:
        1) Nothing happens.  Obviously, our services aren't
           required, and we should stop.  We look silly, need
           to disappear quietly, but still Usenet benefits.
           This didn't happen.
        2) Something happens, but, the results are so confused
           that no sense can be made of it.  This is the outcome
           that I really dreaded, because this whole exercise
           would have been a waste of effort.
           Fortunately, this didn't happen.
        3) Usenet starts rapidly going downhill, and there's no
           sign of anything evolving to fill the gap.  If this
           happens, Usenet 1 is doomed, and it would be far better
           that we simply stop, quit prolonging the agony, and
           stop wasting everyone's time.
           This didn't happen either - we're seeing greatly heightened
           deployment and interest in filtering, and would expect to
           see more work and committment towards things like Usenet 2.
        4) Usenet starts going downhill, but things start to happen
           to counteract it.  The situation should continue to be
           monitored, but, it's possible that cancels can be eased
           up on, and/or, periodically resumed to continue the evolutionary
           pressure yet still give systems some breathing room to deploy
           solutions.
           I think it's pretty clear that this did happen, and will
           continue to happen.
As for continued spam cancelling:
I think it's fair to say that the moratorium is effectively over
as of now.  This was our planned restart date...
Many major despammers have indicated that they will not be
resuming - at least temporarily.  Thus, it's not clear that the
volume of spam cancels will increase much.
It's probably best that regional newsgroup enforcement resume
immediately.  It's done on a somewhat different basis than global
despamming, and is more effective in their own areas than we could
ever be.
Cosmo will probably eliminate his delay.  This will certainly
be noticable.  I would hope that he would continue to have the
delay at his disposal, so as situations warrant, he might
reintroduce it to apply further evolutionary pressure as necessary.
People with groups in alt that they care about should immediately
start considering moving somewhere else.  The death of alt is
obviously coming.  It's time to save what you can.  But we do
have to be prepared for when the spammers, like the rats that
they are ;-), desert that ship, and try to infest the rest of
Usenet.  I would suggest that merely moderating your group
may not be good enough - move now, it's going to be safer in the
long run.
This is what I will be doing:
I encourage other despammers to chime in with what they plan to do.
I will turn despamming back on for the regional that I am involved with.
I will be leaving mainstream despamming off for the immediate
future.  It's not helping in alt, and it's not really necessary
elsewhere at this moment.
I will be turning my NoCeMs off for alt groups - possibly all
alt groups.  Alt is gone, it's time to quit wasting bandwidth
in yet more futile effort to save it.
ISP and user notifications will continue.
I will continue to keep a close watch on spam volumes and help
push continued deployment of filters and/or other solutions such
as Usenet 2, NNTP extensions etc.
As situations warrant, such as major upswings in spam in non-alt
groups, or group bombs, I will turn despamming back on, probably
focused on individual situations only.  We need to be prepared
for spammers attempting to infest other areas now that at least
some of the pressure is off them.
I will be shutting my bot almost entirely off for the next week
or two.  Time to relax for a bit, and the bot desperately needs some
rework.
--
Chris Lewis, CyberSheriff (CBC says I am, so it must be true!)
For more information on spam, see 
http://spam.abuse.net/spam
Fight spam, support Rep. Chris Smith's TCPA extension: 
http://www.cauce.org

 ICFAQ 3.0 - Manuale italiano di Internet 
http://goemon.polito.it/icfaq
 <

Adesso capisco perche' lo spam e' aumentato improvvisamente.
Saluti
-- 
Carlo Fusco
Attenzione l'indirizzo e' stato volutamente alterato: rimuovere ".to" 
BE CAREFUL address intentionally forged: remove  ".to"