 followups to spambayes URL please unless you re specifically concerned about some particular bit of email policy for URL ok after much fiddling with and tweaking of etc exim eximNUMBER conf and etc exim local_scan py on URL i am fairly confident that i can start harvesting all incoming email at a moment s notice for the record here s how it all works eximNUMBER conf works almost exactly the same as before if the file etc exim harvest does not exist that is any junk mail condition that can be detected by exim acls access control lists is handled entirely in eximNUMBER conf the message is rejected before it ever gets to local_scan py this covers such diverse cases as message from known spammer reject after every rcpt to command no message id header and NUMBER bit chars in subject both rejected after the message headers body are read the main things i have changed in the absence of etc exim harvest are don t check for NUMBER bit chars in from header the vast majority of hits for this test were bounces from some asian isp the remaining hits should be handled by spamassassin do header sender verification ie ensure that there s a verifiable email address in at least one of from reply to and sender as late as possible because it requires dns lookups which can be slow and can also make messages that should have been rejected merely be deferred if those dns lookups timeout if etc exim harvest exists then the behaviour of all of those acls in eximNUMBER conf suddenly changes instead of rejecting recipients or messages they add an x reject header to the message this header is purely for internal use it records the name of the folder to which the rejected message should be saved and also gives the smtp error message which should ultimately be used to reject the message thus those messages will now be seen by local_scan py which now looks for the x reject header if found it uses the folder name specified there to save the message and then rejects it with the smtp error message also given in x reject currently x reject is retained in saved messages if a message was not tagged with x reject then local_scan py runs the usual virus and spam checks namely my homebrew scan for attachments with filenames that look like windows executables and a run through spamassassin the logic is basically this if virus folder virus else run through spamassassin if score NUMBER NUMBER folder rejected spam elif score NUMBER NUMBER folder caught spam finally local_scan py writes the message to the designated folder by far the biggest folder will be accepted the server handles NUMBER NUMBER incoming messages per day of which maybe NUMBER NUMBER are junk mail oops just realized i haven t written the code that actually saves the message d ohh also haven t written anything to discriminate personal email which i must do sigh finally the big catch waiting until after you ve read the message headers and body to actually reject the message is problematic because certain broken mtas including those used by some spammers don t consider a NUMBERxx after data as a permanent error but keep retrying d ohh this is a minor annoyance currently where a fair amount of stuff is rejected at rcpt to time but in harvest mode everything with the exception of people probing for open relays will be rejected at data time so i have cooked up something called the asbl or automated sender blacklist this is just a berkeley db file that maps sender_ip sender_address to an expiry time when local_scan rejects a message from sender_ip sender_address for whatever reason including finding an x reject header added by an acl in eximNUMBER conf it adds a record to the asbl with an expiry time NUMBER hours in the future meanwhile there s an acl in eximNUMBER conf that checks for records in the asbl if there s a record for the current sender_ip sender_address that hasn t expired yet we reject all recipients without ever looking at the message headers or body the downside of this from the point of view of corpus collection is that if some jerk is busily spamming URL one smtp connection per address we will most likely only get one copy this is a win if you re just thinking about reducing server load and bandwidth but i m not sure if it s helpful for training spam detectors tim happy harvesting greg greg ward gward python net URL budget s in the red let s tax religion dead kennedys 