Filter out speakers with zero replies that got caught up in the ballots aggregation because they have an unconfirmed ballot as reply speaker