SlowGuess/NAACL_2022_OCR huggingface dataset:
Rahul Sharma*
rahul.sharma@usc.edu
Anil Ramakrishna*
aniramak@amazon.com
Ansel MacLaughlin
ammaclau@amazon.com
Anna Rumshisky
arum@cs.uml.edu
Jimit Majmudar
mjimit@amazon.com
Clement Chung
chungcle@amazon.com
avestime@usc.edu
gupra@amazon.com
Machine Learning (ML) systems are getting increasingly popular, and drive more and more applications and services in our daily life. This has led to growing concerns over user privacy, since human interaction data typically needs to be transmitted to the cloud in order to train and improve such systems. Federated learning (FL) has recently emerged as a method for training ML models on edge devices using sensitive user data and is seen as a way to mitigate concerns over data privacy. However, since ML models are most commonly trained with label supervision, we need a way to extract labels on edge to make FL viable. In this work, we propose a strategy for training FL models using positive and negative user feedback. We also design a novel framework to study different noise patterns in user feedback, and explore how well standard noise-robust objectives can help mitigate this noise when training models in a federated setting. We evaluate our proposed training setup through detailed experiments on two text classification datasets and analyze the effects of varying levels of user reliability and feedback noise on model performance. We show that our method improves substantially over a self-training baseline, achieving performance closer to models trained with full supervision.
Artificial Intelligence (AI) and Machine Learning (ML) have become increasingly common in modern society with applications ranging from simple standalone products to complex modules Kaplan and Haenlein (2019). However, this rise has also created growing privacy concerns Papernot et al. (2016); Yeom et al. (2018). Such concerns may affect user willingness the adapt new technologies Guhr et al. (2020). In response, many government agencies across the world have introduced regulations to protect the data-handling rights of their
citizens, such as the European Union's GDPR General Data Protection Regulation and California's CCPA California Consumer Privacy Act.
Federated Learning (FL) is a step in this direction to improve consumer trust, where models are trained without moving data out of client devices. The typical FL approach is to iteratively train local models on edge devices and then propagate them back to a central node in order to update the global model. Most commonly, this is done using Federated Averaging (FedAvg) McMahan et al. (2017), where we take a simple average over the client model parameters. However, in order to update local models on the edge, this setup assumes the presence of labeled user data on each device, which is often not possible. Most prior works do not address this problem, but simulate fully-supervised federated learning by distributing existing labeled datasets across edge devices. In this work, we consider a more realistic scenario, where labels are not available on device. Rather than turning to unsupervised learning as seen in Hard et al. (2018), we instead propose a novel setup to leverage user feedback in order to train the FL model.
In many real world AI applications with direct or indirect human interaction, it is possible to collect either explicit user feedback (e.g., using a voice or a screen-based prompt) or implicit feedback in the form of behavioral cues. For an example of implicit feedback, consider a user interacting with a virtual AI assistant (such as Alexa), who asks to play the song 'Bohemian Rhapsody' from the band Queen. The virtual assistant would interpret the prompt and select a song from its library to play. If the user lets the music play without interruption, this can be viewed as positive feedback, suggesting that the underlying model interpreted the request correctly. On the other hand, if the user interrupts the music and makes a repeat (or different) request, this can be viewed as negative feedback, suggesting that the underlying model prediction was incorrect.
Human Language Technologies, pages 2726 - 2739
In this work, we propose to leverage such feedback in federated model training.
Leveraging Positive and Negative Feedback In our proposed setup, we first train a seed model on a small amount of labeled data on a central node. This mimics the real-world scenario where a small amount of data can be collected and annotated to bootstrap an initial model. We then propagate this weak seed model to each of the clients. On the edge, we use this seed model to make predictions for each user's request. Since the model is trained with limited data, these predictions may be incorrect. To further improve this model performance, we leverage user feedback as an indirect indicator of the predicted label's quality. Since positive user feedback likely indicates that a model prediction is correct, we label examples with positive feedback with the seed model's prediction and add them to the training data. This mirrors the standard self-training setup Rosenberg et al. (2005), where weak models are further trained on a set of their own predictions. When a user gives negative feedback, however, we cannot assign a label to the example, since we only know that the seed model's prediction is wrong. We instead treat these prediction as complementary labels Ishida et al. (2017); Yu et al. (2018) and extend federated model training to use such labels.
Modeling Feedback Noise In realistic human interactions, however, the user may not always provide consistent feedback, making user feedback signal noisy. In the virtual AI assistant example above, if the model predicts a different song from the same band, the user may choose to continue listening without interruption. Similarly, even if the model predicts the correct song, the user may change their mind once the song starts playing and interrupt with a new request. Such user behavior will introduce noise into the feedback signal. In order to assess typical levels of such noise in user feedback, we conduct a pilot study on Amazon Mechanical Turk (Mturk), and evaluate the accuracy of feedback from Mturk users on two different text classification problems. Based on this study, we define a model of user noise defined by two parameters that specify how likely they are to give accurate feedback on both correct and incorrect predictions by the seed model. With this model of user behaviour, we then study the effects of user noise on model performance. We conduct
extensive experiments on two text classification datasets, training FL models on feedback data with varying amounts of user noise simulated using our model. We further experiment with various noise-robustness strategies to mitigate the effect of such noisy labels and present promising results.
The key contributions in this paper are as follows:
1. We propose a new framework for modeling and leveraging user feedback in FL and present a strategy to train supervised FL models directly on positive and negative user feedback. We show that, under mild to moderate noise conditions, incorporating feedback improves model performance over self-supervised baselines.
2. We propose a novel model of user feedback noise and study the effects of varying levels of this noise on model performance.
3. We study the effectiveness of existing noise-robustness techniques to mitigate the effects of user-feedback noise and identify promising directions for future exploration.
2.1 Federated Learning
Federated Learning McMahan et al. (2017) has recently seen a rise in popularity in a number of domains, including natural language processing (NLP) Yang et al. (2018); Ramaswamy et al. (2019); Hard et al. (2018). This is due to growing privacy concerns Papernot et al. (2016); Geyer et al. (2017); Truex et al. (2019), abundance of unlabeled data, and an increase in the computational capacity on edge devices. However, availability of labels on edge (or rather, lack thereof) limits the practical application of FL in most real-world use cases. In this work, we present an extension to FL and show improvements in federated model performance by leveraging user feedback. Recent works have also revealed risks of information leakage from gradients in federated learning, and several techniques have been developed to mitigate this risk (see Zhu et al. (2019), Lyu et al. (2020) and the references there in).
2.2 Learning With User Feedback
User feedback on model behavior provides learning signals which can be leveraged to continuously improve model performance. Using feedback signals
for model training provides robustness to concept and data drifts, as new data is always accompanied with new feedback labels from which to learn. Learning methods that leverage user feedback have been applied to a variety of tasks in NLP, such as semantic parsing Iyer et al. (2017), machine translation Kreutzer et al. (2018) and question answering Kratzwald and Feuerriegel (2019). To our knowledge, however, our work is the first to build a parametric model of user feedback noise and to study how to train federated learning algorithms with noisy user feedback.
2.3 Negative Label Learning
Standard supervised learning operates on data labeled with their true classes. Feedback data from users, however, can also be negative, indicating that the predicted class is wrong. Since the correct class of examples with such negative-feedback is unknown, our proposed method must be able to handle such ambiguity during training. In our work, we label examples with negative feedback with a complementary label corresponding to the predicted class (Ishida et al., 2017). Complementary labels simply specify the category that a given example does not belong to. In our work, we follow the setup of Yu et al. (2018), who propose loss functions to train neural models on biased complementary labels.
2.4 Noise-Robust Learning
When training models on labels derived from noisy user feedback signals, it is important to use a strategy to mitigate the effects of label noise on model performance. One straightforward approach is to use a noise-robust loss function, such as Reverse Cross Entropy Wang et al. (2019) or Mean Absolute Error Ghosh et al. (2017). In our work we follow the noise-robust learning setup of Ma et al. (2020), who present a training strategy that combines two robust loss functions (one active, one passive) to better handle label noise.
3 Modeling User Feedback
We propose a general framework for leveraging user feedback in federated learning. We use text classification as an exemplar task to evaluate this framework, but the proposed setup can be easily applied to other tasks. We use two benchmark text classification datasets: the Stanford Sentiment Treebank dataset (sst2) and the 20 newsgroups dataset
(20news). The sst2 dataset comprises of 11,855 phrases from movie reviews and the corresponding binary sentiment labels. The 20 newsgroup dataset 20news consists of  $18k$  news articles and headers, organized into 20 classes.
3.1 Pilot Study: Real World User Feedback
To understand the dynamics of user feedback noise, we conduct a pilot study using Amazon Mechanical Turk (Mturk). We use text classification on the above two datasets, sst_2 and 20news, as the task of interest. For each dataset, we train a seed model on  $1\%$  of the training data, then run inference with this model to generate pseudo-labels on the remaining  $99\%$  of the training examples. We sample 2000 pseudo-labeled examples from this set, split them into disjoint groups of 40 examples each, and show them to 50 and 40 different MTurk workers for sst_2 and 20news, respectively. For each example, the worker is shown the text of the example and the corresponding predicted pseudo label. The workers are then asked to specify whether the pseudo label is accurate (positive feedback) or not (negative feedback) along with an option to mark 'I Don't Know' in case they find it difficult to decide. Further details about the specific instructions used in our Mturk study can be found in Appendix C. We use the ground truth gold labels in sst_2 and 20news to evaluate the quality of user feedback by computing the proportions of times users gave positive feedback to correct pseudo labels and negative feedback to incorrect ones. The observed average error in feedback is  $33.9\%$  for 20news and  $27.13\%$  for sst2. The higher observed error for 20news is likely due to the fact that 20news is a 20-way topic classification problem with overlapping categories such as 'autos' and 'motorcycles'. We further analyze the collected data using the noisy feedback model described next. Full data will be released with the final draft of the paper.
3.2 Feedback Noise Modeling
Motivated by the observed noisy user behavior above, we propose a parametric noise model using two user-specific Bernoulli random variables parameterized by  $\gamma$  and  $\delta$ , as shown below.
$$
\begin{array}{c} P (\text {p o s i t i v e f e e d b a c k | c o r r e c t p r e d i c t i o n}) = \gamma \\ P (\text {n e g a t i v e f e e d b a c k | i n c o r r e c t p r e d i c t i o n}) = \delta \end{array}
$$
$\gamma$  and  $\delta$  capture the probability of the user accurately providing positive and negative feedback,
respectively. This scheme provides a powerful tool to model user noises of various types - by varying the values of these two parameters, we can simulate various user feedback behaviors. For instance, highly reliable users can be simulated by choosing  $\gamma \sim 1$  and  $\delta \sim 1$  while adversarial users can be simulated by choosing  $\gamma \sim 0$  and  $\delta \sim 0$ . Similarly, users that provide consistently positive feedback can be simulated by selecting  $\gamma \sim 1$  and  $\delta \sim 0$ , and vice versa.
Figure 1: Distribution of noise parameters  $\gamma$  and  $\delta$  for annotators on Mturk for 20news and sst2 dataset.
For each worker in our MTurk study, we estimate the noise parameters  $\gamma$  and  $\delta$  using the MLE estimators described in Appendix B. We show the distributions over the estimated noise parameters for each worker in Figure 1, which highlights several characteristics of the user noise. We find that parameters vary across users and across datasets. Many workers have high values for both  $\gamma$  and  $\delta$  (upper right quadrant in the plot), especially for the sst2 dataset. In such cases, user noise is relatively low. Some workers have  $\gamma \sim 1$  and  $\delta \sim 0$ , suggesting that they provide positive feedback with very high probability, regardless of the correctness of the pseudo label. Similarly, we also observe some points with higher  $\delta$  but  $\gamma$  close to 0, suggesting that these workers provide negative feedback with high probability. Since we only recruited reliable worker from Mturk ( $95\%+$  approval rate and  $5000+$  approved HITs, see Appendix C), we do
not see any workers in the adversarial or extremely-high noise scenarios (lower-left quadrant in the plot). Finally, we also observe that the workers in the sst2 dataset are more concentrated towards the right top corner while, for the 20news dataset, they are relatively spread out. This can be attributed to the inherent difficulty of the two datasets - sst2 is an easier 2-class sentiment classification dataset, while 20news is a more difficult news-classification dataset with 20 (sometimes overlapping) classes.
4.1 Federated Self-Training
Given a training dataset  $D_{\mathrm{t}} = \{x_i, y_i\}$ , we divide it into 3 parts: a training split  $D_{\mathrm{s}} \subset D_{\mathrm{t}} : |D_{\mathrm{s}}| = k|D_{\mathrm{t}}|$ ,  $k \ll 1$ , used to train the seed model; a validation split  $D_{\mathrm{v}} \subset D_{\mathrm{t}} : |D_{\mathrm{v}}| = v|D_{\mathrm{t}}|$ ,  $v < 1$  and an unlabeled split  $D_{\mathrm{u}} = D_{\mathrm{t}} - (D_{\mathrm{s}} \bigcup D_{\mathrm{v}})$ . We assume that the examples in  $D_{\mathrm{s}}$  and  $D_{\mathrm{v}}$  have gold labels available for training, which mimics the real-world situation where a small amount of data can be annotated to bootstrap the model training. We treat  $D_{\mathrm{u}}$  as the unlabeled dataset which is available on edge. We initialize the seed model  $f_{\mathrm{s}}(x)$  by training on  $D_{s}$  using standard cross entropy loss. After convergence, this model,  $f_{\mathrm{s}}(x)$ , is deployed to the edge devices to start federated training. In order to simulate a real-world federated learning setup, we distribute the examples from  $D_{\mathrm{u}}$  among  $N$  edge clients following a skewed distribution, described in detail in §5. The dataset on each client  $n$  is labeled  $D_{u}^{n}$  where  $n \in [1, N]$ . The seed model on device  $j$  then makes predictions on its corresponding client-specific dataset  $D_{u}^{j}$ . Since the edge model does not have access to gold labels for training, there are only two potential sources of information it can learn from. First, its own predictions,  $\rho_{i}$ , which we call pseudo labels:
$$
\rho_ {i} = \arg \max  \left(f _ {\mathrm {s}} \left(x _ {i}\right)\right): i \in D _ {u} ^ {n} \tag {1}
$$
Labeling an example  $x_{i} \in D_{u}^{n}$  with  $\rho_{i}$  is typically referred to as self-training, a commonly-used semi-supervised training setup. However, in our setup, there is also a second source of information: user feedback to each  $\rho_{i}$ . We assume that users give binary (positive or negative) feedback to each  $\rho_{i}$ . We can thus use this feedback to validate or reject each  $\rho_{i}$ , generating label  $\rho_{i}$  when the feedback is positive and  $\bar{\rho}_{i}$  when the feedback is negative. Then, with these new user-feedback-labeled datasets on each edge device, we can follow the standard FL
training, further training a copy of the initial global model on each edge device, then propagating each local model back to the global server for aggregation. Our overall setup used for federated learning with user feedback is shown in Figure 2.
Figure 2: Overview of our federated learning setup with user feedback. a) Federated learning with a central cloud server and several client devices. b) Local training at a particular client with user feedback to improve pseudo labels.
4.2 User Feedback Simulation
As discussed in §1, in the real world, users provide feedback on predictions made by deployed models. However, large-scale collection of user feedback in a deployed FL application is an expensive endeavor with no publicly-available datasets. In this work, we instead devise a novel framework to simulate realistic noisy user feedback on publicly-available, human-annotated data, and defer the task of real world deployment to future work. Specifically, when we distribute the unlabeled dataset  $D_{u}$  across the  $N$  client devices, we also send along the true gold label for each example  $x_{i}$ . For each  $x_{i} \in D_{u}^{n}$ , we then simulate feedback by comparing the model prediction  $\rho_{i}$  to its underlying gold label. These gold labels are only used to simulate user feedback – they are not used for edge model training. Specifically, we create two new pseudo-labeled datasets corresponding to positive  $(D_{pos}^{n})$  and negative feedback  $(D_{neg}^{n})$  from each device's dataset  $D_{u}^{n}$ . For a given sample  $x_{i} \in D_{u}^{n}$ , we assign it to the positive feedback set  $D_{pos}^{n}$  with probability  $\gamma$  if the corresponding pseudo label  $\rho_{i}$
is correct and  $1 - \delta$  if  $\rho_{i}$  is incorrect. Similarly, we assign a sample to the negative feedback set  $D_{neg}^{n}$  with probability  $1 - \gamma$ , if  $\rho_{i}$  is correct and  $\delta$  if  $\rho_{i}$  is incorrect. A pseudo-code detailing our strategy to simulate user feedback is shown in Algorithm 1.
4.3 Federated Learning with Feedback
For examples with positive user feedback, since we have user confirmation that the pseudo-label  $\rho_{i}$  is correct, we directly incorporate  $\rho_{i}$  into model training as if they were ground-truth. We use the standard categorical cross entropy (CCE) loss function similar to the seed model:
$$
\operatorname {l o s s} _ {\text {p o s}} = - \sum_ {i \in D _ {\text {p o s}} ^ {n}} \rho_ {i} \log \left(f \left(x _ {i}\right)\right) \tag {2}
$$
where  $f(x_{i})$  represents the posterior probability distribution for sample  $i$  and  $\rho_{i}$  is overloaded to depict the one-hot representation of the pseudo label for sample  $i$ . On the other hand, negative feedback signifies that the sample does not belong to the class  $\rho_{i}$ . Although the user feedback does not indicate which class these samples ought to be, we do acquire information for what the model should not do. Thus we can assume that these are complementary labels, denoted as  $\bar{y}_{i} = \rho_{i}$ . To incorporate these in our model training, we adapt the complementary learning methods introduced by Yu et al. (2018), in which the authors model the complementary posterior probability distribution,  $P(\bar{Y} = d|X)$  as a function of true class posterior distribution,  $P(Y = c|X)$  and the transition probability matrix  $\mathbf{Q}$ , where  $q_{cd}$  is an entry in the matrix  $\mathbf{Q}$  with  $c$  and  $d$  are class labels, following:
$$
q _ {c d} = \left\{ \begin{array}{l l} P (\bar {Y} = d | Y = c) & c \neq d \\ 0 & c = d \end{array} \right. \tag {3}
$$
$$
P (\bar {Y} = d | X) = \sum_ {c \neq d} P (\bar {Y} = d | Y = c)
$$
$$
P (Y = c \mid X) \tag {4}
$$
We estimate the transition probability matrix  $\mathbf{Q}$  using the validation set  $D_{v}$  and the initial seed model  $f_{\mathrm{s}}(x)$ . To compute  $\mathbf{Q}_{c}$ , the transition probability distribution for the class  $c$ , we average the posterior probability of those samples with gold label  $c$ , but are incorrectly predicted by the model. Given this, we set  $q_{cc} = 0$  and renormalize the
vector as shown in Equation 5.
$$
\mathbf {Q} _ {c:} = \frac {1}{K} \sum_ {k \in D _ {c}} \frac {f _ {s} \left(x _ {k}\right)}{1 - f _ {s} \left(x _ {k}\right) _ {i}}: K = | D _ {c} | \tag {5}
$$
$$
D _ {c} \subset D _ {v}: a r g m a x (f (x _ {k})) \neq c; y _ {k} = c
$$
Using the estimated transition matrix, and the posterior distribution for the true class, we estimate the posterior distribution for the complementary class, following Equation 3. We then use the pseudo labels as complementary labels and train the model with the objective function:
$$
\operatorname {l o s s} _ {\text {n e g}} = - \sum_ {i \in D _ {\text {n e g}} ^ {n}} \rho_ {i} \log (\mathbf {Q}. f (x _ {i})) \tag {6}
$$
here, we overload  $\rho_{i}$  to depict the one-hot representation of the pseudo label for sample  $i$ .
Our overall model is trained to jointly optimize the loss functions from positive feedback and negative feedback. Inspired by Kim and Kim (2020) where the contribution of negative learning is slowly increased during training, we use a scheduler to weigh the two loss functions, ensuring that the positive label learning component has higher weight in the early epochs, gradually increasing the weight for negative label learning as training proceeds. Specifically, at each client we optimize the following objective:
$$
l o s s _ {r e g} = (1 - \alpha) * l o s s _ {p o s} + \alpha * l o s s _ {n e g} (7)
$$
where,  $\alpha = 1 - p^t$ ,  $t$  denotes the current epoch in the training process and  $p \in (0,1)$ , which was selected using a held out validation set.
4.4 Noise-Robust Loss Functions
Though user feedback provides a valuable learning signal to train our models on edge, it can be noisy. As noted in §3.2, we expect two behaviors from noisy users: if the user provides incorrect positive feedback, we have incorrect true labels in  $D_{pos}^{n}$ . Similarly, if the user provides incorrect negative feedback, we have samples in the  $D_{neg}^{n}$  with incorrect complementary labels. Since we use cross entropy loss functions for training on both positive and negatively labeled data points, our model is prone to overfitting to noisy samples Zhang and Sabuncu (2018) since they would have lower posteriors (forcing the algorithm to put more emphasis on them during training). This necessitates some form of noise mitigation in our model training.