fixes to the requirements file, some reverts in the crawler, and an installer script