semi-automated crawler