Abstract: Deep neural networks (DNNs) can fit (or even over-fit) the training data very well. If a DNN model is trained using data with noisy labels and tested on data with clean labels, the model may perform poorly. This paper studies the problem of learning with noisy labels for sentence-level sentiment classification. We propose a novel DNN model called NetAb (as shorthand for convolutional neural Networks with Ab-networks) to handle noisy labels during training. NetAb consists of two convolutional neural networks, one with a noise transition layer for dealing with the input noisy labels and the other for predicting 'clean' labels. We train the two networks using their respective loss functions in a mutual reinforcement manner. Experimental results demonstrate the effectiveness of the proposed model.

Introduction: It is well known that sentiment annotation or labeling is subjective BIBREF0. Annotators often have many disagreements. This is especially so for crowd-workers who are not well trained. That is why one always feels that there are many errors in an annotated dataset. In this paper, we study whether it is possible to build accurate sentiment classifiers even with noisy-labeled training data. Sentiment classification aims to classify a piece of text according to the polarity of the sentiment expressed in the text, e.g., positive or negative BIBREF1, BIBREF0, BIBREF2. In this work, we focus on sentence-level sentiment classification (SSC) with labeling errors. As we will see in the experiment section, noisy labels in the training data can be highly damaging, especially for DNNs because they easily fit the training data and memorize their labels even when training data are corrupted with noisy labels BIBREF3. Collecting datasets annotated with clean labels is costly and time-consuming as DNN based models usually require a large number of training examples. Researchers and practitioners typically have to resort to crowdsourcing. However, as mentioned above, the crowdsourced annotations can be quite noisy. Research on learning with noisy labels dates back to 1980s BIBREF4. It is still vibrant today BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12 as it is highly challenging. We will discuss the related work in the next section. This paper studies the problem of learning with noisy labels for SSC. Formally, we study the following problem. Problem Definition: Given noisy labeled training sentences $S=\lbrace (x_1,y_1),...,(x_n,y_n)\rbrace $, where $x_i|_{i=1}^n$ is the $i$-th sentence and $y_i\in \lbrace 1,...,c\rbrace $ is the sentiment label of this sentence, the noisy labeled sentences are used to train a DNN model for a SSC task. The trained model is then used to classify sentences with clean labels to one of the $c$ sentiment labels. In this paper, we propose a convolutional neural Network with Ab-networks (NetAb) to deal with noisy labels during training, as shown in Figure FIGREF2. We will introduce the details in the subsequent sections. Basically, NetAb consists of two convolutional neural networks (CNNs) (see Figure FIGREF2), one for learning sentiment scores to predict `clean' labels and the other for learning a noise transition matrix to handle input noisy labels. We call the two CNNs A-network and Ab-network, respectively. The fundamental here is that (1) DNNs memorize easy instances first and gradually adapt to hard instances as training epochs increase BIBREF3, BIBREF13; and (2) noisy labels are theoretically flipped from the clean/true labels by a noise transition matrix BIBREF14, BIBREF15, BIBREF16, BIBREF17. We motivate and propose a CNN model with a transition layer to estimate the noise transition matrix for the input noisy labels, while exploiting another CNN to predict `clean' labels for the input training (and test) sentences. In training, we pre-train A-network in early epochs and then train Ab-network and A-network with their own loss functions in an alternating manner. To our knowledge, this is the first work that addresses the noisy label problem in sentence-level sentiment analysis. Our experimental results show that the proposed model outperforms the state-of-the-art methods.

Question: What is the performance of the model?
Answer: Experiment 1: ACC around 0.5 with 50% noise rate in worst case - clearly higher than baselines for all noise rates
Experiment 2: ACC on real noisy datasets: 0.7 on Movie, 0.79 on Laptop, 0.86 on Restaurant (clearly higher than baselines in almost all cases)