Don't retry failed (404, 403) requests

Adds an in-memory cache for failed requests.