updating project to allow for crawling html data as well as json api