Fix channel deletion on remote clients (Fixes #171)

Refactor some of the channel removal code to make channel removal work again.