added comments/ deleted old code in crawler