It's possible to do so in two ways: 1. Reconnect by pressing the "Connect" button again. 2. Disconnect the speaker.  Both methods will disconnect that speaker from the system without affecting the others.