ENG-5505: Fix rejoin during rebalance