Rule based format agnostic crawler