holoirc: fix setup of activity when serverfragment is slow to be added