Question: How does the ensemble annotator extract the final label?
Answer: First preference is given to the labels that are perfectly matching in all the neural annotators., In case two out of three context models are correct, then it is being checked if that label is also produced by at least one of the non-context models., When we see that none of the context models is producing the same results, then we rank the labels with their respective confidence values produced as a probability distribution using the $softmax$ function. The labels are sorted in descending order according to confidence values. Then we check if the first three (case when one context model and both non-context models produce the same label) or at least two labels are matching, then we allow to pick that one. , Finally, when none the above conditions are fulfilled, we leave out the label with an unknown category.