Refactor code and increase crawler stability for multiple pages