lots of server/latency related fixes [skip ci]