some improvements in crawler