Instead of using inheritance to give the crawler threads a shared, global robots rule-set, I created a static class that they can all access but do not need to instantiate.  So they all share the same rules!