initPeers before Remove
Log peer delete error instead of panic
Try removeFromGroup when delete from owner peer failed