test crawler finished and fixed crawler bugs