Some tinkering to try and remove the audio discontinuity thingy.
Basically increased buffer in AudioEndpointRemote (serverside).
Changed client (pyaudio) to be callback driven.
Did not seem to help any though. But both changes are necessary anyway..