Don't call get_cache if we have no intention of calling set_cache

From #6731.