SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\alpha^ {*} = n _ {3} / \left(n _ {1} + n _ {3} + n _ {5}\right)
$$
$$
\beta^ {*} = n _ {2} / \left(n _ {2} + n _ {4} + n _ {6}\right)
$$
$$
\gamma^ {*} = n _ {1} / \left(n _ {1} + n _ {3} + n _ {5}\right)
$$
$$
\delta^ {*} = n _ {4} / \left(n _ {2} + n _ {4} + n _ {6}\right)
$$
The value of the log-likelihood function at the above critical point, i.e.,  $\log \mathcal{L}(\alpha^{*},\beta^{*},\gamma^{*},\delta^{*}\mid X)$  is positive which suggests that it is the optimizer of the optimization problem in (12).
C Details on MTurk study
Figure 3 shows the instruction page used to guide the workers on Mturk. Since our goal here was to simulate real user feedback for AI systems, we designed the prompt to mimic a situation where users provide their judgements on the accuracy of machine predictions on a given task. Each assignment page had 40 questions for the 20news task (50 for sst2), with an example question shown in Figure 4. For a real world application of this setting, we can imagine an email categorization model deployed on end-user email clients which automatically classifies incoming emails to a predefined class. The user would approve (select Accurate to above question) if the categorization was correct, reject or make correction (select Inaccurate to the question) or take no action. This closely follows the federated user feedback scenario described in our experiments with users explicitly providing positive or negative feedback.
We recruited highly reliable annotators on Mturk by selecting past approval rate as  $95\%+$  and number of past approved tasks as  $5000+$ . The average time for each task was 30 minutes, and the annotators were paid  $\$7$ for completing the task which is above the US federal minimum hourly wage, given the average time for task completion. Note that we did not place any geographic restrictions on the annotators, nor reject any partial submissions, despite stating as such in the instruction sheet, as they were few in number.
In Figure 5, we show the error in user feedback computed against gold labels for all the users. We also show the distribution of positive and negative responses for all the users. As evident from the figure, users provide positive feedback in majority cases. This behavior is expected since the initial model's accuracy for 20news is  $59.14\%$  and for sst2 is  $77.37\%$ ; since a the majority of the pseudo labels are correct predictions, we expect mostly positive feedback from the users.
Please read all instructions listed below, before you start the task.
- Our world is full of interactions between humans and AI systems. We interact with customer service chatbots, with search engines, with personal assistants, among many others. Such interactions happen in our homes, our cars, and pretty much everywhere. But machines don't always get it right. In this task, we ask you to imagine that you requested the machine to assign an email body to a specific topic, such as gun politics, baseball, or motorcycles.
- We will then show you what the machine returned and ask to verify if it was an accurate prediction or not.
- There 20 possible topics the machine can assign your email to, and they may sometimes be overlapping. Try to identify the most precise label that can be assigned to the email. We will regularly show you the 20 labels as you scroll through the questions to help with the task.
- There are 40 examples in this HIT. To get paid, you need to answer all questions before you hit the submit button.
- We wish to get answers from different Mturkers for different tasks without overlaps and we have created separate HITs for each of these, please avoid answering more than one HIT.
Figure 3: Instruction page with guidelines for Mturk annotators.
Input text	Lines: 3 I have 16MB of memory on my 386SX. I have been running Windows without a swap file for several months. Will Mathcad 4.0 be happy with this or insist on a swap file?
Machine prediction	pc hardware
What do you think about the machine prediction? Is it
$\bigcirc$  Accurate
Figure 4: Example annotation task
Figure 5: User feedback behavior of clients in Mturk case study. A & C: Incorrect feedback(\%) for all the clients for 20news and sst2. B & D: Distribution of negative and positive feedback for each client in 20news and sst2