lots of crawler fixes