Tweaks to make crawler run in non-test mode again