IServerFactory is never a registered service

IServerManager is registered when needed

Or the caller may provide context.ServerFactory instance when an
override is needed