Handle case where server is not started successfully, @kbenne take a look to make sure this is ok