Tidy up imports in AbstractServerFactory

Many orphans as their usages have moved to `AbstractHandlerFactory`
instead.
Tidy up imports in DefaultServerFactory

Many orphans due to their usages moving to `DefaultHandlerFactory`.