Apply the restrict and include rules to recaching invalid content, so that we don't recache invalid content unless we want to
Don't recache already crawled paths or paths that are about to be crawled
Merge pull request #3 from jwinter/master

Recaching changes