C# web crawler build for speed and flexibility