changed spider stopping logic to incorporate reposts