jparise/stale — pull request #5 by dhellmann (2013-03-28 00:23:29+00:00)
Title: Ignore errors from specific domains

Some domains, such as nytimes.com, don't support crawlers
(or at least not this one), and always return a 500 or
404 error unless the user is logged in and has a valid
cookie. This changeset adds an option for specifying domains
to ignore and changes the worker to check the URL against a
list of "problem domains" before handling any errors.

Change-Id: I3fd4b670bd5ec4c348bb3fab979a924b5483cf1c
Signed-off-by: Doug Hellmann