[minor] Start with time to connect as initial latency.

This fixes 2 important issues:

1. When we're reconnecting we don't set an insanely high latency as our old ping
   messages were never send.

2. It gives the latency it's initial value. A connect usually involves some package
   thats send back and forth so it gives us an idea on the performance of the
   initial connection.