Experiments	In our experiments we aim to answer the following research questions (RQs): RQ1: How does the number of training iterations in the BERT language model finetuning stage influence the ATSC end-task performance? At what point does performance start to improve, when does it converge? RQ2: When trained in-domain, what ATSC end-task performance can be reached through fully exploitet finetuning of the BERT language model? RQ3: When trained cross-domain in the special case of domain adaptation, what ATSC end-task performance can be reached if BERT language model finetuning is fully exploitet?

Experiments ::: Results Analysis	The results of our experiments are shown in fig:acc-dep-lmiterations and tab:results respectively. To answer RQ1, which is concerned with details on domain-specific language model finetuning, we can see in fig:acc-dep-lmiterations that first of all, language model finetuning has a substantial effect on ATSC end-task performance. Secondly, we see that in the laptops domain the performance starts to increase at about 10 Mio. finetuned sentences. This is an interesting insight as one would expect a relation closer to a logarithmic curve. One reason might be that it takes many steps to train knowledge into the BERT language model due to its vast amount of parameters. The model already converges at around 17 Mio. sentences. More finetuning does not improve performance significantly. In addition, we find that different runs have a high variance, the standard deviation amounts to about $1\%$ in accuracy, which justifies averaging over 9 runs to measure differences in model performance reliably.

Here, domain-specific finetuning could probably bring significant performance improvements. Another interesting direction for future work would be to investigate cross-domain behavior for an additional domain like hotels, which is more similar to the restaurants domain. Here, it could be interesting to find out if the shared nature of these domain would results in more confusion or if they would behave synergetically.

Experiments ::: Compared Methods	We compare in-domain results to current state of the art methods, which we will now describe briefly. SDGCN-BERT  explicitly models sentiment dependencies for sentences with multiple aspects with a graph convolutional network. This method is current state-of-the-art on the SemEval 2014 laptops dataset. AEN-BERT  is an attentional encoder network. When used on top of BERT embeddings this method performs especially well on the laptops dataset. BERT-SPC  is BERT used in sentence-pair classification mode. This is exactly the same method as our BERT-base baseline and therefore, we can cross-check the authors results. BERT-PT  uses multi-task fine-tuning prior to downstream classification, where the BERT language model is finetuned jointly with a question answering task. It performs state-of-the-art on the restaurants dataset prior to this paper. To our knowledge, cross- and joint-domain training on the SemEval 2014 Task 4 datasets has not been analyzed so far. Thus, we compare our method to two very strong baselines: BERT and XLNet.

We think the reason for this might be that syntactical relationships between the aspect-target and the phrase expressing sentiment polarity as well as knowing the sentiment-polarity itself are sufficient to solve the ATSC task in many cases. For the joint-training case, we find that combining both training datasets improves performance on both test sets. This result is intuitive, as more training data leads to better performance if the domains do not confuse each other. Interesting for the joint-training case is that the BERT-ADA Joint model performs especially strong when measured by the Macro-F1 metric. A reason for this might be that the SemEval 2014 datasets are imbalanced due to dominance of positive label. It seems like through finetuning the language model on both domains the model learns to classify the neutral class much better, especially in the laptops domain.

To answer RQ2, which is concerned with in-domain ATSC performance, we see in tab:results that for the in-domain training case, our models BERT-ADA Lapt and BERT-ADA Rest achieve performance close to state-of-the-art on the laptops dataset and new state-of-the-art on the restaurants dataset with accuracies of $79.19\%$ and $87.14\%$, respectively. On the restaurants dataset, this corresponds to an absolute improvement of $2.2\%$ compared to the previous state-of-the-art method BERT-PT. Language model finetuning produces a larger improvement on the restaurants dataset. We think that one reason for that might be that the restaurants domain is underrepresented in the pre-training corpora of BERTBASE. Generally, we find that language model finetuning helps even if the finetuning domain does not match the evaluation domain. We think the reason for this might be that the BERT-base model is pre-trained more on knowledge-based corpora like Wikipedia than on text containing opinions.

Another finding is that BERT-ADA Joint performs better on the laptops dataset than BERT-ADA Rest, although the unique amount of laptop reviews are the same in laptops- and joint-corpora. We think that confusion can be created when mixing the domains, but this needs to be investigated further. We also find that the XLNet-base baseline performs generally stronger than BERT-base and even outperforms BERT-ADA Lapt with an accuracy of $79.89\%$ on the laptops dataset. To answer RQ3, which is concerned with domain adaptation, we can see in the grayed out cells in tab:results, which correspond to the cross-domain adaption case where the BERT language model is trained on the target domain, that domain adaptation works well with $2.2\%$ absolute accuracy improvement on the laptops test set and even $3.6\%$ accuracy improvement on the restaurants test set compared to BERT-base. In general, the ATSC task generalizes well cross-domain, with about 2-$3\%$ drop in accuracy compared to in-domain training.

Related Works ::: Architecture Improvements for Aspect-Target Sentiment Classification	The datasets typically used for Aspect-Target Sentiment Classification are the SemEval 2014 Task 4 datasets  for the restaurants and laptops domain. Unfortunately, both datasets only have a small number of training examples. One common approach to compensate for insufficient training examples is to invent neural architectures that better model ATSC. For example, in the past a big leap in classification performance was achieved with the use of the Memory Network architecture , which uses memory to remember context words and explicitly models attention over both the target word and context. It was found that making full use of context words improves their model compared to previous models  that make use of left- and right-sided context independently.  proposed Attention Encoder Networks (AEN), a modification to the transformer architecture. The authors split the Multi-Head Attention (MHA) layers into Intra-MHA and Inter-MHA layers in order to model target words and context differently, which results in a more lightweight model compared to the transformer architecture. Another recent performance leap was achieved by , who model dependencies between sentiment words explicitly in sentences with more than one aspect-target by using a graph convolutional neural network.

The validation accuracy converges after about 3 epochs of training on all datasets, but training loss still improves after that. It is important to note that all our results reported are the average of 9 runs with different random initializations. This is needed to measure significance of improvements, as the standard deviation in accuray amounts to roughly $1\%$ for all experiments, see fig:acc-dep-lmiterations.

The second variant consists of Aspect-Target Extraction (ATE) followed by Aspect-Target Sentiment Classification (ATSC). ATE is a sequence labeling task, where terms like “dumplings” are detected. In the second step, ATSC, the sentiment polarity associated to the aspect-target is determined. In our example the correct result is the tuple ("dumplings", "positive"). In this work, we focus on ATSC. In the last years, specialized neural architectures ,  have been developed that substantially improved modeling of this target-context relationship. More recently, the Natural Language Processing community experienced a substantial shift towards using pre-trained language models , , ,  as a base for many down-stream tasks, including ABSA , , . We still see huge potential that comes with this trend, this is why we approach the ATSC task using the BERT architecture. As shown by , for the ATSC task the performance of models that were pre-trained on general text corpora is improved substantially by finetuning the model on domain-specific corpora — in their case review corpora — that have not been used for pre-training BERT, or other language models. We extend the work by Xu et al.

by further investigating the behavior of finetuning the BERT language model in relation to ATSC performance. In particular, our contributions are: The analysis of the influence of the amount of training-steps used for BERT language model finetuning on the Aspect-Target Sentiment Classification performance. The findings on how to exploit BERT language model finetuning enables us to achieve new state-of-the-art performance on the SemEval 2014 restaurants dataset. The analysis of cross-domain adaptation between the laptops and restaurants domain. Adaptation is tested by finetuning the BERT language model self-supervised on the target-domain and then supervised training on the ATSC task in the source-domain. In addition, the performance of training on the combination of both datasets is measured.

Conclusion	We performed experiments on the task of Aspect-Target Sentiment Classification by first finetuning a pre-trained BERT model on a domain specific corpus with subsequent training on the down-stream classification task. We analyzed the behavior of the number of domain-specific BERT language model finetuning steps in relation to the end-task performance. With the findings on how to best exploit BERT language model finetuning we were able to train high performing models, which one of even performs as new state-of-the-art on SemEval 2014 Task 4 restaurants dataset. We further evaluated our models cross-domain to explore the robustness of Aspect-Target Sentiment Classification. We found that in general, this task transfers well between the laptops and the restaurants domain. As a special case we ran a cross-domain adaptation experiments, where the BERT language model is specifically finetuned on the target domain. We achieve significant improvement over unadapted models, a cross-domain adapted model performs even better than a BERT-base model that is trained in-domain. Overall, our findings reveal promising directions for follow-up work. The XLNet-base model performs strongly on the ATSC task.

For the restaurants domain more reviews are available, we sampled $10,000,000$ sentences to have a sufficient amount of data for fully exploitet language model finetuning. In order to compensate for the smaller amount of finetuning data in the laptops domain, we finetune for more epochs, 30 epochs in the case of the laptops domain compared to 3 epochs for the restaurants domain, so that the BERT model trains on about 30 million sentences in both cases. This means that 1 sentence can be seen multiple times with a different language model masking. We also create a mixed corpus to jointly finetune both domains. Here, we sample 1 Mio. restaurant reviews and combine them with the laptop reviews. This results in about 2 Mio. reviews that are finetuned for 15 epochs. The exact statistics for the three finetuning corpora are shown in the top of tab:datasets. To be able to reproduce our finetuning corpora, we make the code that is used to generate them available online.

Introduction	Sentiment Analysis (SA) is an active field of research in Natural Language Processing and deals with opinions in text. A typical application of classical SA in an industrial setting would be to classify a document like a product review into positve, negative or neutral sentiment polarity. In constrast to SA, the more fine-grained task of Aspect Based Sentiment Analysis (ABSA) ,  aims at finding both the aspect of an entity like a restaurant and the sentiment associated with this aspect. It is important to note that ABSA comes in two variants. We will use the sentence “I love their dumplings” to explain these variants in detail. Both variants are implemented as a two-step procedure. The first variant is comprised of Aspect-Category Detection (ACD) followed by Aspect-Category Sentiment Classification (ACSC). ACD is a multilabel classification task, where a sentence can be associated with a set of predefined aspect categories like "food" and "service" in the restaurants domain. In the second step, ACSC, the sentiment polarity associated to the aspect is classified. For our example-sentence the correct result is (“food”, “positive”).

They show that their architecture performs particularly well if multiple aspects are present in a sentence.

Methodology ::: Cross-Domain Adaptation	As a special case of cross-domain Training we expect performance to be optimal if $D_{LM} = T$. This is the variant of Domain Adaptation and is written as $T \rightarrow S \rightarrow T.$

Methodology ::: Domain Adaptation through Language Model Finetuning	In academia, it is common that the performance of a machine learning model is evaluated in-domain. This means that the model is evaluated on a test set that comes from the same distribution as the training set. In real-world applications this setting is not always valid, as the trained model is used to predict previously unseen data. In order to evaluate the performance of a machine learning model more robustly, its generalization error can be evaluated across different domains, i.e. cross-domain. Additionally, the model itself can be adapted towards a target domain. This is known as Domain Adaptation, which is a special case of Transductive Transfer Learning in the taxonomy of . Here, it is typically assumed that supervised data for a specific task is only available for a source domain $S$, whereas only unsupervised data is available in the target domain $T$. The goal is to optimize performance of the task in the target domain while transferring task-specific knowledge from the source domain. If we map this framework to our challenge, we define Aspect-Target Sentiment Classification as the transfer-task and BERT language model finetuning is used for domain adaptation.

Methodology ::: BERT Language Model Finetuning	As the first step of our procedure we perform language model finetuning of the BERT model using domain-specific corpora. Algorithmically, this is equivalent to pre-training. The domain-specific language model finetuning as an intermediate step to ATSC has been shown by . As an extension to their paper we investigate the limits of language model finetuning in terms of how end-task performance is dependent on the amount of training steps. The training input representation for language model finetuning consists of two sequences $s_A$ and $s_B$ in the format of $"\textrm {[CLS]} \ s_{A} \ \textrm {[SEP]} \ s_{B} \ \textrm {[SEP]}"$, where [CLS] is a dummy token used for downstream classification and [SEP] are separator tokens.

Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification	Aspect-Target Sentiment Classification (ATSC) is a subtask of Aspect-Based Sentiment Analysis (ABSA), which has many applications e.g. in e-commerce, where data and insights from reviews can be leveraged to create value for businesses and customers. Recently, deep transfer-learning methods have been applied successfully to a myriad of Natural Language Processing (NLP) tasks, including ATSC. Building on top of the prominent BERT language model, we approach ATSC using a two-step procedure: self-supervised domain-specific BERT language model finetuning, followed by supervised task-specific finetuning. Our findings on how to best exploit domain-specific language model finetuning enable us to produce new state-of-the-art performance on the SemEval 2014 Task 4 restaurants dataset. In addition, to explore the real-world robustness of our models, we perform cross-domain evaluation. We show that a cross-domain adapted BERT language model performs significantly better than strong baseline models like vanilla BERT-base and XLNet-base. Finally, we conduct a case study to interpret model prediction errors.

Related Works	We separate our discussion of related work into two areas: First, neural methods applied to ATSC that have improved performance solely by model architecture improvements. Secondly, methods that additionally aim to transfer knowledge from semantically related tasks or domains.

Methodology ::: In-Domain Training	ATSC is trained on a domain-specific dataset and evaluated on the test set from the same domain. This can be expressed as $D_{LM} \rightarrow T \rightarrow T,$ where $T$ is our target domain and can be either Laptops or Restaurants. It is expected that the performance of the model is best if $D_{LM} = T$.

Methodology	We approach the Aspect-Target Sentiment Classification task using a two-step procedure. We use the pre-trained BERT architecture as a basis. In the first step we finetune the pre-trained weights of the language model further in a self-supervised way on a domain-specific corpus. In the second step we train the finetuned language model in a supervised way on the ATSC end-task. In the following subsections, we discuss the BERT architecture, how we finetune the language model, and how we transform the ATSC task into a BERT sequence-pair classification task . Finally, we discuss the different end-task training and domain-specific finetuning combinations we employ to evaluate our model's generalization performance not only in-domain but also cross-domain.

Methodology ::: BERT	The BERT model builds on many previous innovations: contextualized word representations , the transformer architecture , and pre-training on a language modeling task with subsequent end-to-end finetuning on a downstream task , . Due to being deeply bidirectional, the BERT architecture creates very powerful sequence representations that perform extremely well on many downstream tasks . The main innovation of BERT is that instead of using the objective of next-word prediction a different objective is used to train the language model. This objective consists of 2 parts. The first part is the masked language model objective, where the model learns to predict tokens, which have been randomly masked, from the context. The second part is the next-sequence prediction objective, where the model needs to predict if a sequence $B$ would naturally follow the previous sequence $A$. This objective enables the model to capture long-term dependencies better. Both objectives are discussed in more detail in the next section. As a base for our experiments we use the BERTBASE model, which has been pre-trained by the Google research team. It has the following parameters: 12 layers, 768 hidden dimensions per token and 12 attention heads. It has 110 Mio.

parameters in total. For finetuning the BERT language model on a specific domain we use the weights of BERTBASE as a starting point.

In terms of on which domain is finetuned on, the full transfer-procedure can be expressed in the following way: Here, $D_{LM}$ stands for the domain on which the language model is finetuned and can take on the values of Restaurants, Laptops or (Restaurants $\cup $ Laptops). The domain for training $D_{Train}$ can take on the same values, for the joint case case the training datasets for laptops and restaurants are simply combined. The domain for testing $D_{Test}$ can only be take on the values Restaurants or Laptops. Combining finetuning and training steps gives us nine different evaluation scenarios, which we group into the following four categories:

Methodology ::: BERT Language Model Finetuning ::: Masked Language Model Objective	The sequences $A$ and $B$ have tokens randomly masked out in order for the model to learn to predict them. The following example shows why domain-specific finetuning can alleviate the bias from pre-training on a Wikipedia corpus: "The touchscreen is an [MASK] device". In the fact-based context of Wikipedia the [MASK] could be "input" and in the review domain a typical guess could be the general opinion word "amazing".

Methodology ::: BERT Language Model Finetuning ::: Next-Sentence Prediction	In order to train BERT to capture long-term dependencies better, the model is trained to predict if sequence $B$ follows sequence $A$. If this is the case, sequence A and sequence B are jointly sampled from the same document in the order they are occuring naturally. Otherwise the sequences are sampled randomly from the training corpus.

Methodology ::: Aspect-Target Sentiment Classification	The ATSC task aims at classifying sentiment polarity into the three classes positive, negative, neutral with respect to an aspect-target. The input to the classifier is a tokenized sentence $s=s_{1:n}$ and a target $t=s_{j:j+m}$ contained in the sentence, where $j < j+m \le n$. Similar to previous work by , we transform the input into a format compatible with BERT sequence-pair classification tasks: $"\textrm {[CLS]} \ s \ \textrm {[SEP]} \ t \ \textrm {[SEP]}"$. In the BERT architecture the position of the token embeddings is structurally maintained after each Multi-Head Attention layer. Therefore, we refer to the last hidden representation of the [CLS] token as $h_{[CLS]} \in \mathbf {R}^{768 \times 1}$. The number of sentiment polarity classes is three.

A distribution $p \in [0,1]^3$ over these classes is predicted using a fully-connected layer with 3 output neurons on top of $h_{[CLS]}$, followed by a softmax activation function where $b \in \mathbf {R}^3$ and $W \in \mathbf {R}^{3 \times 768}$. Cross-entropy is used as the training loss. The way we use BERT for classifying the sentiment polaritites is equivalent to how BERT is used for sequence-pair classification tasks in the original paper .

Methodology ::: Cross-Domain Training	ATSC is trained on a domain-specific dataset and evaluated on the test set from the other domain. This can be expressed as $D_{LM} \rightarrow S \rightarrow T,$ where $S\ne T$ are source and target domain and can be either Laptops or Restaurants.

Methodology ::: Joint-Domain Training	ATSC is trained on both domain-specific datasets jointly and evaluated on both test sets independently. This can be expressed as $D_{LM} \rightarrow (S \cup T) \rightarrow T,$ where $S\ne T$ are source- and target domain and can either be Laptops or Restaurants.

Experiments ::: Datasets for Classification and Language Model Finetuning	We conduct experiments using the two SemEval 2014 Task 4 Subtask 2 datasets  for the laptops and the restaurants domain. The two datasets contain sentences with multiple marked aspect terms that each have a 3-level sentiment polarity (positive, neutral or negative) associated. In the original dataset the conflict label is also present. Here, conflicting labels are dropped for reasons of comparability with . Both datasets are small, detailed statistics are shown in tab:datasets. For BERT language model finetuning we prepare three corpora for the two domains of laptops and restaurants. For the restaurants domain we use Yelp Dataset Challenge reviews and for the laptops domain we use Amazon Laptop reviews . For the laptop domain we filtered out reviews that appear in the SemEval 2014 laptops dataset to avoid training bias for the test data. To be compatible with the next-sentence prediction task used during fine tuning, we removed reviews containing less than two sentences. For the laptop corpus, $1,007,209$ sentences are left after pre-processing.

Experiments ::: Hyperparameters	We use BERTBASE (uncased) as the base for all of our experiments, with the exception of XLNetBASE (cased), which is used as one of the baseline models. For the BERT language model finetuning we use 32 bit floating point computations using the Adam optimizer . The batchsize is set to 32 while the learning rate is set to $3\cdot 10^{-5}$. The maximum input sequence length is set to 256 tokens, which amounts to about 4 sentences per sequence on average. As shown in tab:datasets, we finetune the language models on each domain so that the model trains a total of about 30 Mio. sentences (7.5 Mio. sequences). For training the BERT and XLNet models on the down-stream task of ATSC we use mixed 16 bit and 32 bit floating point computations, the Adam optimizer, and a learning rate of $3\cdot 10^{-5}$ and a batchsize of 32. We train the model for a total of 7 epochs.

BERT-base  is using the pre-trained BERTBASE embeddings directly on the down-stream task without any domain specific language model finetuning. XLNet-base  is a method also based on general language model pre-training similar to BERT. Instead of randomly masking tokens for pre-training like in BERT a more general permutation objective is used, where all possible variants of masking are fully exploitet. Our models are BERT models whose language model has been finetuned on different domain corpora. BERT-ADA Lapt is the BERT language model finetuned on the laptops domain corpus. BERT-ADA Rest is the BERT language model finetuned on the restaurant domain corpus. BERT-ADA Joint is the BERT language model finetuned on the corpus containing an equal amount of laptops and restaurants reviews.

Related Works ::: Knowledge Transfer for Aspect-Target Sentiment Classification Analysis	Another approach to compensate for insufficient training examples is to transfer knowledge across domains or across similar tasks.  proposed Multi-Granularity Alignment Networks (MGAN). They use this architecture to transfer knowledge from both an aspect-category classification task and also across different domains. They built a large scale aspect-category dataset specifically for this.  transfer knowledge from a document-level sentiment classification task trained on the amazon review dataset . They successfully apply pre-training by reusing the weights of a Long Short Term Memory (LSTM) network  that has been trained on the document-level sentiment task. In addition, they apply multi-task learning where aspect and document-level tasks are learned simultaneously by minimizing a joint loss function. Similarly,  introduce a multi-task loss function to simultaneously optimize the BERT model's  pre-training objectives as well as a question answering task. In contrast to the methods described above that aim to transfer knowledge from a different source task like question answering or document-level sentiment classification, this paper aims at transferring knowledge across different domains by finetuning the BERT language model.

Question: What are the performance results?
Answer: results that for the in-domain training case, our models BERT-ADA Lapt and BERT-ADA Rest achieve performance close to state-of-the-art on the laptops dataset, new state-of-the-art on the restaurants dataset with accuracies of $79.19\%$ and $87.14\%$, respectively.