Remove scrapyd from project and improve tests
Removing all usage of scrapyd and the communicator