Fix observer removal logic