Copy cache to a temp file, so upgrader can later delete it