add user auth and index for crawler
add pagination