update to site crawler input. added in dependencies files for site crawler