More changes to crawler, still many bugs and design flaws :(