fix bugs, prepare to integrate with crawler