Remove bans related to the server on server.remove()

Signed-off-by: Ian Macalinao <ianmacalinao@gmail.com>