CAMEL-7146: Fixed aggregator when completion size is 1, eg when completed asap. Should not call remove as we did not add.