Fast path when server address is IP.