Do not stat local file if it does not exist; better cleanup of tmp files if something goes wrong
Allow local mirror to set robots policy