Revert "Cleaned up some code of the crawler"

This reverts commit f149d4a3de59d12ecaae2711f39ff8e12f29e413.