It turns out if we don't specify the hostname for the web socket server, it works on both localhost and remote devices at the same time.