SlowGuess/NAACL_2022_OCR huggingface dataset:
Orith Toledo-Ronen, Matan Orbach, Yoav Katz, Noam Slonim IBM Research
{oritht, matano, katz, noams}@il.ibm.com
Targeted Sentiment Analysis (TSA) is a central task for generating insights from consumer reviews. Such content is extremely diverse, with sites like Amazon or Yelp containing reviews on products and businesses from many different domains. A real-world TSA system should gracefully handle that diversity. This can be achieved by a multi-domain model – one that is robust to the domain of the analyzed texts, and performs well on various domains. To address this scenario, we present a multi-domain TSA system based on augmenting a given training set with diverse weak labels from assorted domains. These are obtained through self-training on the YELP reviews corpus. Extensive experiments with our approach on three evaluation datasets across different domains demonstrate the effectiveness of our solution. We further analyze how restrictions imposed on the available labeled data affect the performance, and compare the proposed method to the costly alternative of manually gathering diverse TSA labeled data. Our results and analysis show that our approach is a promising step towards a practical domain-robust TSA system.
Customer reviews of products and businesses provide insights for both consumers and companies. They help companies understand customer satisfaction or guide marketing campaigns, and aid consumers in their decision-making. Sentiment analysis plays a central role in the analysis of such material, by aiming to understand the sentiment expressed in a review document or in a single review sentence (Liu, 2012). Beyond these high-level trends, identifying the sentiment towards a specific product feature or an entity is important. Such a fine-grained analysis includes the key task of Targeted Sentiment Analysis (TSA), aimed at detecting sentiment-bearing terms in texts and classifying the sentiment towards them. For example, in the
sentence "The room was noisy, but the food was tasty," the targets are room and food with negative and positive sentiments, respectively. Our focus in this work is on TSA of user reviews in English.
A real-world TSA system has to successfully process diverse data. From toothbrushes to phones, airline companies to local retailers, the online content today covers a broad range of reviews in many domains. Ideally, a system for such a multi-domain scenario should be able to cope with inputs from any domain, those that were seen during training, and, perhaps more importantly, those that were not.
To the best of our knowledge, this work is the first to pursue TSA in a multi-domain setup, intending to support input from multiple unknown domains. Many previous works have used the indomain setup of training and testing on data from the same domain (e.g. Li et al. (2019b)). Newer works focus on the cross-domain setup, yet most have explored a pairwise evaluation of training on one source domain and evaluating on a single known target domain (e.g. Rietzler et al. (2020); Gong et al. (2020)).
Broadly, multi-domain learning (Joshi et al., 2012) includes training and evaluation using data from multiple domains (e.g. Dredze and Crammer (2008); Qin et al. (2020); Dai et al. (2021b)). Sometimes, it is assumed that the input texts are accompanied by a domain label (e.g. Joshi et al. (2012)). Here, we do not assume a domain label is given – this has the advantage of allowing easier practical use of our model, without having to specify the domain as part of the input. In other cases, evaluation is limited to domains represented in training, or otherwise performed in a zero-shot setup only on unseen domains (Wang et al., 2020). Our system handles both cases simultaneously, processing data from domains well-represented in the training data as well as from unseen domains.
For practical reasons, implementing a multidomain system with a single model that can handle
Human Language Technologies, pages 2751 - 2762
all domains is desirable. This can save valuable resources such as memory or GPUs, which are in high demand by contemporary language models (LMs). For example, it is impractical to expect that an online service providing TSA analysis will have a per-domain model, each keeping its many parameters in memory, along with perhaps a set of pre-allocated GPUs. Our goal is therefore to have a single multi-domain model that performs well on both seen and unseen domains. This is reminiscent of works in multilingual NLP that develop a single model that handles multiple languages (e.g. M-BERT released by Devlin et al. (2019), Liang et al. (2020), Toledo-Ronen et al. (2020)).
A possible approach to our setting is training on a diverse TSA dataset, potentially encompassing many of the domains that the system is applied to. However, obtaining such a dataset is a challenge. The existing TSA datasets are limited in their diversity, and the collection of a new large scale diverse TSA dataset is complex (Orbach et al., 2021).
The road we take is therefore based on augmenting a TSA dataset of limited diversity with assortment of weak labels, through self-training – one of the earliest ideas for utilizing unlabeled data in training (Chapelle et al., 2009). To show that our approach is feasible, we performed an extensive empirical evaluation with several LMs that were fine-tuned with labeled data from the SEMEVAL dataset of Pontiki et al. (2014) (henceforth SE). This dataset is limited to two domains: restaurants or laptops. Each model went through several self-training iterations and evaluated on three TSA publicly available datasets: SE, the MAMS dataset of restaurant reviews (Jiang et al., 2019), and the YASO dataset of open-domain reviews (Orbach et al., 2021).
As part of our evaluation, we created two new TSA resources. The first is an annotation layer on top of the YASO dataset, specifying the domain of each review. This allows a per-domain evaluation providing insights on the performance of seen and unseen domains. The second resource is a set of manually annotated TSA reviews, which can be an ad-hoc diverse TSA training set, an alternative to the proposed method. We show that even in the presence of such data in training our approach is valuable. Both resources are available online. $^{1}$
In summary, the main contributions of this work are: (i) the first exploration of TSA in a multi
domain setup; (ii) demonstrating the feasibility of multi-domain TSA by an extensive evaluation on three datasets and the use of self-training; (iii) the release of additional TSA resources: a new annotation layer for the YASO dataset, and a set of fully annotated reviews.
TSA The TSA task has been extensively studied in different scenarios. Some works considered it as a pipeline of two subtasks: (i) aspect-term extraction (TE) for identifying target terms in texts (e.g. Li et al. (2018); Xu et al. (2018)), and (ii) aspect-term sentiment classification (SC) for determining the sentiment towards a given target term (e.g. Dai et al. (2021a); Li et al. (2019c); Wang et al. (2018)). Full TSA systems may combine these building blocks by running TE and then SC in a pipeline. Others, like our system, use a single engine that provides an end-to-end solution to the whole task, and may be based on pre-trained language models (e.g. Li et al. (2019b); Phan and Ogunbona (2020)) or a generative approach (Yan et al., 2021; Zhang et al., 2021). In a cross-domain setup, TSA research includes Chen and Qian (2021) on TE, Rietzler et al. (2020) on SC, Wang and Pan (2020); Pereg et al. (2020) for joint TE and opinion term extraction and Gong et al. (2020) for the full TSA task. In contrast with our setup, these works all evaluate on one known domain.
Domain Adaptation A plethora of domain adaptation (DA) methods have been developed for handling data from domains that are under-represented in training. Several DA variants exist, of which the most common one handles a single known target domain. For sentiment analysis, DA is especially important, as sentiment baring words tend to differ between domains (Ruder et al., 2017). One promising DA approach is adjusting a given LM to a target domain using pre-training tasks performed on unlabeled data from that domain (Xu et al., 2019; Rietzler et al., 2020; Zhou et al., 2020). Another recently proposed direction of DA explored self-training for sentiment analysis (e.g. Liu et al. (2021)).
Self Training At the core of our approach is the iterative process of self-training. This methodology has been successfully applied for varied research problems, e.g. object detection (Rosenberg et al., 2005), parsing (McClosky et al., 2006), handwritten digit recognition (Lee, 2013) and image clas
github.com/IBM/yaso-tsa
sification (Zou et al., 2019) (see also the survey by Triguero et al. (2015)). Since the emergence of pre-trained LMs, several works have explored finetuning these models through self-training. Some examples are works on sentiment and topic classification (Yu et al., 2021), negation detection (Su et al., 2021), toxic span detection (Suman and Jain, 2021), text classification (Karamanolakis et al., 2021) and machine translation (Sun et al., 2021).
Our self-training approach augments a given TSA training set with weak-labels (WL) generated from a large multi-domain corpus. The process (depicted in Figure 1) starts by training an initial TSA model on that given training data. Then, that model produces TSA predictions on a large unlabeled corpus of diverse reviews. Finally, some of the predictions are selected and added as weak labels to the original training set. A new model is then trained with the augmented data, applied to produce new predictions on the unlabeled data, and the whole process (detailed below) can repeat for several iterations.
3.1 TSA Engine
We consider TSA as a sequence tagging problem, where the model predicts a discrete label for each token of the input sequence. The possible labels are: positive (P), negative (N) or none (O). The first two labels represent tokens that are part of a sentiment target, and the O label represents all other non-target tokens. For example, given "Here is a nice electric car", the desired output is the target electric car, identified from the output word-level sequence (O, O, O, O, P, P). During inference, for each sub-word piece within the input text, the labels scores outputted by the transformer model are converted into probabilities by applying softmax, and the highest probability label is selected. The sub-word pieces predictions within each word are then merged by inducing the label of the first word piece with sentiment on the other word-pieces. Finally, consecutive word sequences having the same label (P or N) constitute one predicted target.
Our tagging scheme falls under the category of a unified tagging scheme (Li et al., 2019a) with IO labels. Previous works with a unified scheme used the more complex IOBES labels (Li et al., 2019a,b), where the B and E labels designate the beginning and end of a target, respectively, and S represents a single token target. Observing that the labeled
data rarely includes two adjacent targets, the B and E labels were omitted (following Breck et al. (2007)). The S label was excluded since in practice tokenization was to sub-word pieces, making the prediction of a single S label redundant.
3.2 Unlabeled Data Set
We use the YELP reviews data to create the weakly-labeled dataset for training. We start the process by extracting  $2M$  sentences from the YELP corpus<sup>2</sup>. The corpus contains the text of the review documents and a list of business categories that correspond to each review. The reviews were initially selected at random, and then some reviews were removed by two conditions: reviews that are rated as not useful (with useful=0) and reviews of businesses with no business categories. For each review, we assigned a single representative domain based on its business categories. The domain was determined by the first match between the review's categories and a predefined list of domains constructed from the categories in the corpus ordered by their popularity.
Following the document-level filtering, each review was split into sentences, and the sentences were further filtered by: 1) length: only sentences with 10-50 words were selected; and 2) sentiment: at least one sentiment word should appear in the sentence. For the sentiment filter, we used a general-purpose lexicon – the Opinion Lexicon (Hu and Liu, 2004) that was automatically expanded by an SVM classifier and filtered as described in Bar-Haim et al. (2017). From that lexicon, we took all the sentiment words with score  $S$  with confidence threshold of  $|S| > 0.7$ , resulting with 7497 sentiment words.
Finally, the representative domain of each review was assigned to all its selected sentences. Overall, we identified 18 different domains in the  $2M$  extracted sentences, as shown in Table 1. We can see that  $60\%$  of the extracted data is from restaurants reviews, but the other  $40\%$  of the data cover a variety of other domains.
3.3 Generating Weak labels
The process, depicted in Figure 1, starts by training a model on TSA labeled data (henceforth, the LD model), followed by iteratively generating TSA weak labels by self-training. The initial LD model is used for predicting TSA target spans and semi
2yelp.com/dataset
Domain	Sentences	Domain	Sentences
Restaurants	1,195,156	Entertainment	47,618
Food	109,278	Bars	31,449
Beauty&Spas	106,023	Pets	26,679
Services	102,471	Local Flavor	10,688
Travel	92,600	Education	6,561
Shopping	87,224	Nightlife	3,855
Automotive	66,107	Television	2,170
Health	60,768	Religious	1,468
Active Life	49,094	Media	791
Table 1: Data extracted from the YELP corpus with total of 2M sentences in 18 domains.
ments on the unsupervised data. Each prediction is associated with a score  $S$ . We use it as a confidence score and select a subset of the sentences according to the following recipe: 1) targets: sentences with targets that have confidence  $S > 0.9$  are selected if all other targets in the same sentence have  $S <= 0.5$ . The high-confidence targets are added to the TSA weak labels and the other predictions are ignored; 2) non-targets: sentences with no predictions or if all the predicted targets have score  $S <= 0.5$  are selected and all the predictions are ignored. To limit the amount of this part of the data, these sentences are randomly selected from  $10\%$  of the data. 3) domain balancing: the number of sentences per domain is limited to  $20k$  for each part of the data – for sentences with targets and for those with no identified targets. This creates a balance between the representation of different domains in the data. Without balancing, about half of the selected data is from restaurants and other domains are under-represented.
The selected sentences (those with TSA weak labels and those with no targets) are then added to the labeled data, and a new TSA model is finetuned and used for TSA prediction and sentence selection over the entire unsupervised dataset in the next iteration. We repeated the process of WL generation and model training 3 times and used the model from the third iteration for evaluation. The total number of sentences in the WL data generated from the 2M sentences extracted from YELP is about 280k. This number depends on the initial LD model and on the number of iterations performed.
4.1 Evaluation Data
YASO In Orbach et al. (2021), we presented the YASO TSA dataset comprising of user reviews from multiple sources. This dataset covers reviews from many domains, and is thus a good choice for
multi-domain evaluation. While YASO allows an assessment on diverse reviews, its data is unbalanced between domains, thus biasing a standard evaluation towards the more common domains. A per-domain evaluation is therefore complementary, and can help validate that a model performs well on all domains, not just the common ones. Such an evaluation can also aid in discerning between performance on domains that are well-represented in the labeled data and ones that are unseen, thus verifying that the evaluated model performs well in both cases.
To facilitate such a per-domain evaluation, we augmented YASO with a domain label for each of its annotated reviews. The assigned labels were produced automatically, when possible, or otherwise they were manually set by one of the authors. Since YASO contains annotated reviews from multiple sources, the assigned label depended on the source: reviews taken from the Stanford Sentiment Treebank (Socher et al., 2013; Pang and Lee, 2005) were assigned the movies domain label. Reviews from the OPINOSIS source (Ganesan et al., 2010) were assigned a label of electronics, automotive or hotels, based on the topic provided in that corpus for each review. For example, reviews on transmission_toyota_camry_2007 were assigned to automotive. In the YELP source, each review is associated with a list of business categories. These categories were used as domain labels: we manually selected 8 prominent categories as domains, and automatically matched the reviews to the domains using the category lists. Reviews matched to multiple categories were manually examined and assigned the most fitting domain from the matched categories. Texts from the AMAZON source (Keung et al., 2020) were manually read and labeled.
Finally, the assigned domain labels were categorized into: restaurants (with 400 sentences), electronics (412), hotels (161), automotive (144), movies (500) and other (596). This extra annotation layer of the YASO evaluation data is available online (see §1). As suggested in Orbach et al. (2021), YASO is used solely for evaluation.
MAMS Jiang et al. (2019) collected the MAMS dataset over restaurant reviews. In MAMS, each sentence has at least two targets<sup>3</sup> annotated with different sentiments. The sentiments are either positive, negative or neutral. To match our setup, the neutral labels were removed from these data. The
3Called aspect terms in Jiang et al. (2019).
Figure 1: Weak labels generation and TSA modeling process.
500 sentences of the MAMS test set serve as an additional evaluation set.
SE Pontiki et al. (2014) created the popular SE dataset of restaurants and laptops reviews. We follow the standard split of SE into two sets with 6072 training sentences and 1600 test sentences. In each set, the sentences are balanced between the two domains. As in MAMS, the neutral labels were removed, as well as the mixed sentiment labels.
4.2 Language Models
The following four pre-trained LMs were used in our experiments:
BERT-B (Devlin et al., 2019) The BERT-base uncased model with 110M parameters.
BERT-MLM To adjust BERT-B to user reviews and sentiment analysis, we further pre-train it on the Masked Language Model (MLM) task, using the  $2M$  review sentences extracted from YELP (see §3.2). Our masking includes two randomly selected sets: (i)  $15\%$  of the words in each sentence, as in BERT-B; (ii)  $30\%$  of the sentiment words in each sentence. The sentiment words are taken from the union of two sentiment lexicons, one of Bar-Haim et al. (2017) (with a confidence threshold of 0.7), and the other created by Toledo-Ronen et al. (2018) (with a confidence threshold of 0.5, yielding 445 words not present in the first lexicon). Our masking of sentiment words is similar to the method used by Zhou et al. (2020), yet we do not use the emoticon masking.
BERT-PT (Xu et al., 2019) A variant of BERT-B post-trained on the MLM and Next Sentence Prediction tasks using YELP data from the restaurants domain, and question answering data.
SENTIX (Zhou et al., 2020) A sentiment-aware language model for cross-domain sentiment analysis. This model was pre-trained with reviews from
Yelp and Amazon, using an MLM task that randomly masks sentiment words, emoticons, and regular words.
4.3 Experimental setting
Training Our fine-tuning used a cross-entropy loss, the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 3e-5 and epsilon of 1e-8. The training process was running with batch size of 32 on 2 GPUs with a maximum of 15 epochs and early stopping with min delta of 0.005. In each experiment,  $20\%$  of the training set sentences were randomly sampled and used as a development set. The optimized metric on this set was the overall token  $F_{1}$  classification rate.
Evaluation For each experiment, we trained 10 models with different random seeds. Then, the per-domain performance metrics were computed for each run, and averaged for a final per-domain result (mean and standard deviation). These per-domain results were macro-averaged to obtain the overall performance on each dataset. As evaluation metrics we report the precision (P), recall (R), and  $F_{1}$  (mean and std), of exact match predictions.
Before showing the multi-domain results that are the focus of this work, we present the in-domain performance of our system on the widely-used SE evaluation data. These results, summarized in Table 2, serve as a sanity check for our system on a well-known benchmark in a well-explored setup.
Explicitly, several single-domain models were created by fine-tuning each pre-trained LM with training data from one SE domain, either restaurants (R) or laptops (L). These models, denoted  $\mathrm{SE}_{R / L}$ , were evaluated on test data from the same domain they were trained on. For BERT-B, the results of this evaluation (top row of Table 2) were inline with previous works (cf. Wang et al. (2021)).
For each LM, Table 2 further shows the results
huggingface.co/activebus/BERT-PTrest
LM	Train Set	Restaurants	Laptops
P	R	F1	P	R	F1
BERT-B	SE R/L	67.7	77.3	72.1 ± 0.8	55.9	65.8	60.4 ± 1.3
SE R/L+WL	74.0	75.3	74.6 ± 0.5	63.7	63.4	63.6 ± 0.8
BERT-MLM	SE R/L	70.9	81.7	75.9 ± 0.7	57.4	64.8	60.8 ± 1.3
SE R/L+WL	76.0	79.8	77.8 ± 0.3	62.4	63.3	62.8 ± 0.8
BERT-PT	SE R/L	71.6	81.4	76.1 ± 0.8	58.1	67.2	62.3 ± 1.5
SE R/L+WL	78.4	77.1	77.7 ± 0.6	63.6	65.0	64.2 ± 0.9
SENTIX	SE R/L	70.4	80.2	74.9 ± 0.7	60.3	70.6	65.0 ± 1.1
SE R/L+WL	76.3	78.4	77.4 ± 0.3	65.7	67.5	66.6 ± 0.7
Table 2: In-domain results on SE comparing fine-tuning of four  $LMs$  with in-domain labeled data  $(\mathbf{S}\mathbf{E}_{R / L})$  and with self-training  $(\mathbf{S}\mathbf{E}_{R / L} + \mathbf{W}\mathbf{L})$ .
with added WL data  $(\mathrm{SE}_{R / L} + \mathrm{WL})$  , created using the corresponding  $\mathrm{SE}_{R / L}$  model on the diverse YELP corpus. Interestingly, in all cases augmenting the training set with these WL improves results over the models trained without such data.
4.5 Multi-Domain Results
For the main evaluation of our approach, we fine-tuned each LM with the full SE training set (with data of both the R and L domains), generated the WL data by self-training starting from the baseline model (SE), and then fine-tuned the final model  $(\mathrm{SE} + \mathrm{WL})$ . Table 3 presents the results obtained with these fine-tuned models, on YASO, MAMS, and SE. In all cases,  $F_{1}$  is improved by employing self-training. For example, with BERT-B, there is a  $10\%$  relative gain in  $F_{1}$  on YASO and MAMS, and a  $3\%$  relative gain on SE. Even with stronger base models such as SENTIX or BERT-PT that incorporate domain knowledge into the language model, we see gains of several points in  $F_{1}$  by adding the WL data. The gain in  $F_{1}$  is mostly due to gain in precision, sometimes at some cost in recall (specifically for MAMS). The variance of  $F_{1}$  across the different training runs is significantly reduced.
Figure 2 further details per-domain results on YASO, showing precision/recall curves for each fine-tuned LM with and without self-training. As above, each curve is the average of 10 per-run curves. In most cases, the self-trained models outperform the initial corresponding fine-tuned SE models. This result is also apparent in Figure 3 for MAMS. Here, although recall is decreased for self-trained models their precision is significantly improved across the entire curve.