generalize crawler class