Refactor crawler dead code

Removing dead code (around 1,000 lines of C) and cleaning up the `gray-crawler` code base makes everyone happier :+1: !