Added some test code to show how to use the crawler on its own by
subclassing it and how to use it with qless