improve crawler implementation without changing external interface.

improve crawler implementation without changing external interface.

use recursive call and async await.