changes to crawler to make off-site use easier