SlowGuess/NAACL_2021_OCR huggingface dataset:
Ismini Lourentzou, Kabir Manghnani, and ChengXiang Zhai. 2019. Adapting sequence to sequence models for text normalization in social media. Proceedings of the International AAAI Conference on Web and Social Media, 13(01):335-345.
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765-4774. Curran Associates, Inc.
Paul Michel and Graham Neubig. 2018. MTNT: A Testbed for Machine Translation of Noisy Text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 543-553.
Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. 2013. Learning with noisy labels. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1, NIPS'13, page 1196-1204. Curran Associates Inc.
Federico Alberto Pozzi, Elisabetta Fersini, Enza Messina, and Bing Liu. 2016. Sentiment analysis in social networks. Morgan Kaufmann.
Ines Rehbein and Josef Ruppenhofer. 2017. Detecting annotation noise in automatically labelled data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1160-1170. Association for Computational Linguistics.
Filipe N Ribeiro, Matheus Araujo, Pollyanna Gonçalves, Marcos Andre Gonçalves, and Fabricio Benevenuto. 2016. SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Science, 5(1):23.
Michael R. Smith and Tony Martinez. 2011. Improving classification accuracy by identifying and removing instances that should be misclassified. In The 2011 International Joint Conference on Neural Networks, pages 2690-2697. IEEE.
Michael R. Smith, Tony Martinez, and Christophe Giraud-Carrier. 2014. An instance level analysis of data complexity. Machine Learning, 95(2):225-256.
J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. 2012. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323-332.
Ana Valdivia, Emiliya Hrabova, Iti Chaturvedi, M. Victoria Luzón, Luigi Troiano, Erik Cambria, and Francisco Herrera. 2019. Inconsistencies on TripAdvisor reviews: A unified index between users and Sentiment Analysis Methods. Neurocomputing, 353:3-16.
Jason D. Van Hulse, Taghi M. Khoshgoftaar, and Haiying Huang. 2007. The pairwise attribute noise detection algorithm. Knowledge and Information Systems, 11(2):171-190.
A. Vinciarelli. 2005. Noisy text categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(12):1882-1895.
Xingyou Wang, Weijie Jiang, and Zhiyong Luo. 2016. Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2428-2437. The COLING 2016 Organizing Committee.
Yongfeng Zhang. 2015. Incorporating Phrase-level Sentiment Analysis on Textual Reviews for Personalized Recommendation. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining - WSDM '15, pages 435-440. ACM Press.
Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis C. M. Lau. 2015. A C-LSTM neural network for text classification. CoRR, abs/1511.08630.
Xingquan Zhu and Xindong Wu. 2004. Class Noise vs. Attribute Noise: A Quantitative Study. Artificial Intelligence Review, 22(3):177-210.
A.1 Examples of Reviews
Tables 4 and 5 show some examples of hard instances labeled by our human classifier. All the code and data is publicly available at https://github.com/karenstemartins/NAACL2021.
class label (l_i)	Example
discrepant	"Figgis's film doesn't match its reach."
(Positive)
mixed	"Pleasant but dull formula film." (Negative)
factual	"Without trivializing the disease, the film challenges AIDS' stigma (albeit for heterosexuals) at a moment when it was still considered a death sentence." (Positive)
contextual	"Disheveled tripe pieced together with the good intentions." (Negative)
undefined	"More interesting as history, re-written, than as the moral parable this true story became."
(Positive)
Table 4: Examples of hard instances from experts.
A.2 Regular Reviews
Table 6 shows real examples of misclassified regular reviews with their original polarities given by their authors. The first and last review contain
class label (li)	Example
discrepant	“The actors try their best with the lines they are given, but the "movie about a real bank robbery" is on auto-pilot most of the time. It greatly resembles a 70’s film by letting the characters drive the story. As a result there's a lot of dialog. But its not very interesting dialog. It is an instantly forgettable film.” (Positive)
mixed	“I think the director did an incredible job. I loved the way it was shot. The scifi world they created was also awesome. But I think the story was way too subtle and wasn't clear enough.” (Positive)
factual	“(...) The 1953 film about a provincial old couple's pilgrimage to the big city provokes sympathy for the mother and father, who are so frail, so gentle, and yet are treated so badly by their urbanized son and daughter. (...)” (Positive)
contextual	“Only go if you're interested in seeing Bening's new facelift.” (Negative)
undefined	“Wow, can’t believe the critics on this one.” (Positive)
some “irony” that is clear to humans, but they are created using words with their opposite polarity of the final score. The second review contains positive and negative points of the movie, but humans can easily identify what side has the most weight.
Table 5: Examples of hard instances from regular users.
Examples
"Michael Bay may think that special effects can substitute for good acting and a good story, but that does not fly around here." (Negative)
"From the first moment of Superman till the very end scene Lex luthor this is a true comic book movie adaption. True there are few CGI errors, but "Nothing is perfect in this world" and this is just a movie." (Positive)
"The trailer was promising to me; I expected it to be a really good movie, but instead it was "meh". I didn't really like Cruz; it was heartwarming how Lightning McQueen made a tribute to Doc at the end, but the trailer made it seem action packed; it wasn't as good as I expected." (Negative)
Table 6: Examples of misclassified regular reviews.
We further investigate these patterns by using SHAP (Lundberg and Lee, 2017), which is a game theoretic approach to explain the output of deep learning models and designed to understand the most important words for the machine classifiers. Figure 3 shows the result for the last review in Table 6. The words are plotted in descending order according with their importance. Note that, all listed words have a positive polarity when they are analyzed separately. As a result, their combination
contributes to the classifier misclassify the review.
Figure 3: SHAP plot for the last review in Table 6.
A.3 Machine Classifier
A.3.1 Third Part Material
The code of the three machine classifiers used in this work are publicly available in the Internet. CNN-GRU and BERT were published by their authors and C-LSTM by researchers who used this method in their work (Elaraby and Abdul-Mageed, 2018). We made small modifications in the codes so they are able to process our movie reviews data. We also created a log module to register all the results and changed the final output layer to a sigmoid function, since our problem is a binary classification. We also made BERT use the Keras library just to facilitate our comparisons, but this is not a necessary step to reproduce our results. The link to each repository is listed below:
C-LSTM: https://github.com/ EngSalem/TextClassification_ Off_the_shelf;
- CNN-GRU: https://github.com/ultimate010/crrn;
- BERT: https://github.com/google-research/bert;
A.3.2 Model Training
To train the machine classifiers, we randomly generated two balanced partitions of our data with the same size, one for experts and other for regular users. Each training dataset contains 4,398 positive and 4,398 negative reviews, for a total of 8,796 reviews. It is important to note that these datasets do not contain any review labeled by the human classifier. After that, we performed a 5-fold cross-validation to choose the best hyperparameters for our data. The set of hyperparameter configurations we tested were the same used in the original articles (Wang et al., 2016), (Zhou et al., 2015) and (Devlin et al., 2019). Since the BERT architecture
is very simple, it has only a single hyperparameter, the batch size, for which we tested values of 16, 32 and 64. For C-LSTM, we tested layers with 100, 150 and 200 filters, and filters of size 2, 3 and 4, memory dimensions of size 100, 150 and 200, and batch size of 16, 32 and 64. Finally, for CNN-GRU, we tested layers with 100 and 200 filters, filters of size 3 and 4, GRU dimensionality of 100 and 150, pool sizes of 2 and 3, and batch sizes of 16 and 32. To run our experiments, we use a computer with the following configuration: 32 RAM, Intel Core i7 CPU 3.40 GHz and NVIDIA GeForce GTX GPU.
After executing cross-validation, we selected the best hyperparameters for each architecture and type of users comparing their F1-Score. We use 256 words for models trained with expert data and 512 for those trained with regular user data in all architectures. BERT achieved the best results using a batch size of 16 for both user types. For experts, C-LSTM uses a batch size of 32, 100 filters with size 3 in the convolutional layer, and 200 as memory dimension for LSTM. For regular users, the hyperparameters are the same, except in the LSTM layer, where a memory dimension of 100 was used. For experts, CNN-GRU uses 100 filters with size 5 as filter length and 3 as pool size for both CNNs. In the GRU, we used dimensionality of 150 and batch size of 16. For regular users, the differences are that we used a dimensionality of 100 in the GRU layer, size 3 as filter length and 2 as pool size for both CNNs. For both C-LSTM and CNN-GRU the differences in the hyperparameters are explained by the fact that our expert reviews are significantly shorter than the ones wrote by regular users. After selecting the best hyperparameters, we trained two models for each architecture, one for experts and other for regular users. Also, each result reported in the paper is the average of five runs, where for each run the model is trained from start using the whole training dataset. With that, we can measure their parameter sensitivity and calculate confidence intervals for the results. In addition, C-LSTM and CNN-GRU took approximately half a day to train and BERT one day. Finally, we noted that the performance of all three models were not significantly affected by the hyperparameter configurations we tested.