Add ability to temporarily silence checks

This also includes a couple of other minor changes ported from the Twitter repo.