Added more crawler functionality, cleaned up design a little