Smarter handling of errors and backoff in reloading reference; only actually propagate an exception after many failures, or failure on first, forced retrieval