holoirc: fix some subtle bugs

* fix when the activity is not displayed but is still available, the join/part/disconnect events do not come through