----------------------------------- ----------------------------------- --
  ![](media/image1.wmf)                                                   

  ----------------------------------- ----------------------------------- --

Site Anti-Spam Improvements Spring 2005

## 

## What is Spam

Spam is any unwanted, unsolicited email. In some organizations, spam
makes up anywhere from 60 to 90 percent of incoming email.

## Spam Filtering at Fermilab

Fighting spam is a multistage process. All email coming in to Fermilab
is first run through our spam filter machines, called hepa#.fnal.gov.
These machines process the email through a program called SpamAssassin.

SpamAssassin runs a large variety of tests on the email and assigns a
score based on the likelihood that an email is spam. This score and some
other info is stored in extra headers that are inserted into the email.
It does not otherwise modify the email.

From here it passes through two different virus scanners and then on to
the final destination on site. For most users, that will be on the IMAP
servers. The IMAP servers are configured by default to move any email
that SpamAssassin has decided may be spam to a folder called \'Tagged
Spam\'. If you don\'t see your \'Tagged Spam\' folder, you may need to
subscribe to it.

### What Spam Assassin Does

There are two main kinds of tests that SpamAssassin uses. The first type
is regular expression tests that check for various words and phrases
that are known to occur in spam. This can include anything from whole
phrases like "buy this now" to more esoteric tests like too many \$\'s
or !\'s in a message. There are also tests for things like too many HTML
font changes or known methods of pretending an email came from
someone\'s Outlook program instead of a spam sending program.

The other test type is Remote Black hole Lists (RBL\'s). There are sites
that search for and maintain lists of machines that are known to either
be spammers themselves, or allow spammers to use their servers, or are
hacked or mis-configured to allow relaying of spam. Spam Assassin checks
all received headers in the message against these lists, and if it
passed through a suspicious machine, will increase its spam score.

## Proposed Improvements

###  Bayesian Filtering

SpamAssassin includes a feature called "Bayesian Filtering". This
feature checks for spam by building a database of words found in emails
and assigning a score for each word based on the number of times it has
been seen in spam emails and non-spam emails. When a new message comes
in all its words are looked up in the spam and non-spam word databases
and a "spam probability" is calculated from the individual word scores.
This score will then be added or subtracted to the messages
"traditional" SpamAssassin score.

The drawback to this approach is that it needs regular "training" to
make sure its word databases are up to date. We will address this in two
ways.

First, we will enable a feature called "auto-training". This will take
each message, and if it is a non-spam message with a sufficiently low
score, it will add its words to the non-spam database. It will do the
corresponding training for spam messages with a score above a certain
threshold, which is higher than the normal spam threshold.

The second method will be via feedback from users. We will provide a web
form for submitting messages that have been mis-categorized as spam or
not-spam.

The web form will have two text areas and a submit button. The first
text area will be for cutting and pasting spam messages that were not
labeled as spam, and the second will be for non-spam messages that were
erroneously labeled as spam. Once the submit button is pressed, the
message will be queued for the training program. The user will receive a
thank you message and will be represented with the form for submitting
more messages. There will be no other feedback from the system.

The messages submitted will need to include full headers. The
SpamAssassin training program is smart enough to recognize duplicate
submissions during the training process and will not adversely effect
the word database quality.

The messages that have been submitted as spam and non-spam will be
reviewed in an administrators web form for approval before submitting
them to the learning system to make sure there aren\'t any obvious
mistakes in the samples being fed to the learning program. This may be
automated in the future.

## Future Enhancements

###  Per-User Settings

The next improvement after this will be individualized spam score
thresholds and individualized whitelists and blacklists. This will
require setting up a shared database for the hepa machines to access for
per user information.

###  Auto Whitelist

There is another somewhat similar feature in SpamAssassin called "Auto
Whitelist". With this feature, another address database is maintained
that contains an average of all the spam scores for a particular sender.
Once a senders address has an entry in this database, it will be used to
adjust the spam scores from the sender closer to the "spam score"
average.

This has the effect that if someone who never sends spam for whatever
reason sends an email that ends up scoring high, its score will be
reduced somewhat at the Auto Whitelist test, so its less likely to be
marked as spam.

Conversely, if a known spammer suddenly creates an email that doesn\'t
score as spam, its score will get bumped up somewhat by the Auto
Whitelist test so its more likely to get marked as spam.

## References

\"Paul Graham\'s original proposal for Bayesian
Filtering\":http://www.paulgraham.com/spam.html

\"SpamAssassin Wiki page on Bayesian
filtering\":http://wiki.apache.org/spamassassin/BayesFaq

\"Prototype spam submission
page\":http://www-test.fnal.gov/cgi-bin/teachspam/teachspam.cgi