audio: Fix PulseAudio driver trying to connect to a non-existent server
forever.