fixed up client side a bit plus small changes in crawler