Get the remote endpoint at accept time and use it