sunlightlabs/scout — pull request #494 by konklone (2014-05-03 00:57:20+00:00)
Title: Drastically reduce various admin emails

This pull request consolidates and eliminates various admin email messages, in an attempt to greatly reduce the email Scout sends the administrator.

It *disables* the following types of emails:

* Simple delivery notices, sent every time any delivery happens. For example, "Sent 4 notifications", and the body details which notifications were sent to who. They are rarely helpful, and are voluminous (especially the `email_immediate` ones).
* Notices of new feed subscriptions. Their purpose was to watch for abuse of the feed import feature. There hasn't been any abuse, and that abuse is best monitored without an email for every one.
* Notices of confirmed user accounts. This was nice to see when we first launched the quick-signup feature, but it's not very useful now.
* Daily notices of Google crawler activity. It stops the underlying logging of each Googlebot hit as well. These were useful for a while, but don't justify the hit to database space or response time anymore, especially after optimizing collection pages in various commits and through #493.
* Potential bounce notifications from Postmark's bounce hook. The hook has been removed. We weren't using it anyway (I think I removed it via the Postmark admin area long ago), and it's essentially a double notification even if it were working.

It fixes the following bug that reduces email notifications:

* Accounts we had auto-unsubscribed for a hard bounce or spam complaint, but which had Open States accounts, were getting resubscribed during Open States' nightly sync. This adds a field to users that marks them as `bounced`, and during the Open States sync, we refuse to sync data to any `bounced` users. A test has been added that verifies this behavior.

It aggregates the following types of emails:

* Backfill notices. These occur when, during checking for new items, sudden old items (that have not been seen before) appear in the stream. This indicates turbulence in the remote source, and sending the results may be jarring to the user (and possibly so old as to be useless), so they are not sent, and turned into warnings. Depending on the kind of turbulence, a great many backfill notices may appear. This stores them in the database for a day, and adds a daily cronjob that emails them en masse, and then clears them for the day.
* CourtListener warnings. These occur when, during checking for new court opinions, we get results back from CourtListener that appear to be for another user. We don't send these results, and instead make a warning for them. This has been a recurring problem, that we're working with CourtListener to resolve. But in the meantime, they can appear in large numbers, so this follows the same path as backfill notices, creating a single nightly aggregate email.

Taken together, this should drastically reduce the number of emails sent to the admin, without impacting the admin's ability to see high priority events and analytics.

Emails are still sent on uncaught exceptions, network errors to remote sources during fetch/sync/check/initialize, on detecting orphaned subscriptions, re-initializing uninitialized subscriptions, weekly analytics reports on clicks and user activity, user-initiated one-click unsubscribes, and new user registrations (including users who come through Open States). I may be forgetting some other types of notifications.