prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which models/frameworks do they compare to?
Context 1: Introduction
Deep neural networks (DNNs), in particular convolutional and recurrent neural networks, with huge architectures have been proven successful in wide range of tasks including audio processing such as speech to text [1 - 4], emotion recognition [5 - 8], speech/non-speech (e g , of non-speech include noise, music, etc ,) classification [9 - 12], etc 
Training these deep architectures require large amount of annotated data, as a result, they cannot be used in low data resource scenarios which is common in speech-based applications [13 - 15]  Apart from collecting large data corpus, annotating the data is also very difficult, and requires manual supervision and efforts  Especially, annotation of speech for tasks like emotion recognition also suffer from lack of agreement among the annotators [16]  Hence, there is a need to build reliable systems that can work in low resource scenario 
In this work, we propose a novel approach to address the task of classification in low data resource scenarios  Our approach involves simultaneously considering more than one sample (in this work, two samples are considered) to train the classifier  We call this approach as simultaneous two sample learning (s2sL)  The proposed approach is also applicable to low resource data suffering with data imbalance  The contributions of this paper are:
Proposed approach
The s2sL approach proposed to address low data resource problem is explained in this Section  In this work, we use MLP (modified to handle our data representation) as the base classifier  Here, we explain the s2sL approach by considering two-class classification task 
Data representation
Consider a two-class classification task with INLINEFORM0 denoting the set of class labels, and let INLINEFORM1 and INLINEFORM2 be the number of samples corresponding to INLINEFORM3 and INLINEFORM4 , respectively  In general, to train a classifier, the samples in the train set are provided as input-output pairs as follows  DISPLAYFORM0
where INLINEFORM0 refers to the INLINEFORM1 -dimensional feature vector representing the INLINEFORM2 sample corresponding to INLINEFORM3 class label, and INLINEFORM4 , refers to output label of INLINEFORM5 class  In the proposed data representation format, called simultaneous two sample (s2s) representation, we will simultaneously consider two samples as follows  DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 are the INLINEFORM2 -dimensional feature vectors representing the INLINEFORM3 sample in INLINEFORM4 class and INLINEFORM5 sample in the INLINEFORM6 class, respectively; and INLINEFORM7 , INLINEFORM8 refers to the output labels of INLINEFORM9 and INLINEFORM10 class, respectively 
Hence, in s2s data representation, we will have an input feature vector of length INLINEFORM0 i e , INLINEFORM1 , and output class labels as either INLINEFORM2 or INLINEFORM3   Each sample can be combined with all the samples (i e , with ( INLINEFORM4 ) samples) in the dataset  Therefore, by representing the data in the s2s format, the number of samples in the train set increases to INLINEFORM5 from INLINEFORM6 samples  We hypothesize that the s2s format is expected to help the network not only to learn the characteristics of the two classes separately, but also the difference and similarity in characteristics of the two classes 
Classifier training
MLP, the most commonly used feed forward neural network, is considered as the base classifier to validate our proposed s2s framework  Generally, MLPs are trained using the data format given by eq  INLINEFORM0   But to train the MLP on our s2s based data representation (as in eq  INLINEFORM1 ), the following modifications are made to the MLP architecture (refer to Figure FIGREF4 ) 
We have INLINEFORM0 units (instead of INLINEFORM1 units) in the input layer to accept the two samples i e , INLINEFORM2 and INLINEFORM3 , simultaneously 
The structure of the hidden layer in this approach is similar to that of a regular MLP  The number of hidden layers and hidden units can be varied depending upon the complexity of the problem  The number of units in the hidden layer is selected empirically by varying the number of hidden units from 2 to twice the length of the input layer (i e , 2 to INLINEFORM0 ) and the unit at which the highest performance is obtained are selected  In this paper, we considered only a single hidden layer  Rectified linear units (ReLU) are used for hidden layer 
The output layer will consist of units equal to twice the considered number of classes in the classification task i e, the output layer will have four units for two-class classification task  The sigmoid activation function (not softmax) is used on the output layer units  Unlike regular MLP, we use sigmoid activation units in the output layer, with binary cross-entropy as the cost function, because the output labels in the proposed s2s based data representation will have more than one unit active at a time (not one-hot encoded output) and this condition cannot be handled using softmax function 
As can be seen from Figure FIGREF4 , the output layer in our proposed method has outputs INLINEFORM0 and INLINEFORM1 which corresponds to the outputs associated with the input feature vectors INLINEFORM2 and INLINEFORM3 , respectively  For a two-class classification problem, there will be four units in the output layer and the possible output labels are INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 corresponding to the class labels INLINEFORM8 and INLINEFORM9 , respectively  This architecture is referred to as s2s-MLP  In s2sL, s2s-MLP is trained using the s2s data representation format  Further, the s2s-MLP is trained using adam optimizer 
Classifier testing
Generally, the feature vector corresponding to the test sample is provided as input to the trained MLP in the testing phase and the class label is decided based on the obtained output 
However, in s2sL method, the feature vector corresponding to the test sample should also be converted to the s2s data representation format to test the trained s2s-MLP  We propose a testing approach, where the given test sample is combined with a set of preselected reference samples, whose class label is known a priori, to generate multiple instances of the same test sample as follows  DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 refer to the INLINEFORM2 -dimensional feature vector corresponding to the test sample and the INLINEFORM3 reference sample, respectively  INLINEFORM4 refers to the considered number of reference samples  These reference samples can be selected from any of the two classes 
For testing the s2s-MLP (as shown in Figure FIGREF4 ), each test sample INLINEFORM0 (same as ' INLINEFORM1 ' in (3)) is combined with all the INLINEFORM2 reference samples ( INLINEFORM3 ) to form INLINEFORM4 different instances of the same test sample INLINEFORM5   The corresponding outputs ( INLINEFORM6 ) obtained from s2s-MLP for the INLINEFORM7 generated instances of INLINEFORM8 are combined by voting-based decision approach to obtain the final decision INLINEFORM9   The class label that gets maximum votes is considered as the predicted output label 
Experiments
We validate the performance of the proposed s2sL by providing the preliminary results obtained on two different tasks namely, Speech/Music discrimination and emotion classification  We considered the GTZAN Music-Speech dataset [17], consisting of 120 audio files (60 speech and 60 music), for task of classifying speech and music  Each audio file (of 2 seconds duration) is represented using a 13-dimensional mel-frequency cepstral coefficient (MFCC) vector, where each MFCC vector is the average of all the frame level (frame size of 30 msec and an overlap of 10 msec) MFCC vectors  It is to be noted that our main intention for this task is not better feature selection, but to demonstrate the effectiveness of our approach, in particular for low data scenarios 
The standard Berlin speech emotion database (EMO-DB) [18] consisting of 535 utterances corresponding to 7 different emotions is considered for the task of emotion classification  Each utterance is represented by a 19-dimensional feature vector obtained by using the feature selection algorithm from WEKA toolkit [19] on the 384-dimensional utterance level feature vector obtained using openSMILE toolkit [20]  For two class classification, we considered the two most confusing emotion pairs i e , (Neutral,Sad) and (Anger, Happy)  Data corresponding to Speech/Music classification (60 speech and 60 music samples) and Neutral/Sad classification (79 neutral and 62 sad utterances) is balanced whereas Anger/Happy classification task has data imbalance, with anger forming the majority class (127 samples) and happy forming the minority class (71 samples)  Therefore, we show the performance of s2sL on both, balanced and imbalanced datasets 
All experimental results are validated using 5-fold cross validation (80% of data for training and 20% for testing in each fold)  Further, to analyze the effectiveness of s2sL in low resource scenarios, different proportions of training data, within each fold, are considered to train the system  For this analysis, we considered 4 different proportions i e , INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 of the training data to train the classifier  For instance, INLINEFORM4 means considering only half of the original training data to train the classifier, and INLINEFORM5 means considering the complete training data  5-fold cross validation is considered for all data proportions  Accuracy (in %) is used as a performance measure for balanced data classification tasks (i e , Speech/Music classification and Neutral/Sad emotion classification), whereas the more preferred INLINEFORM6 measure [21] is used as a measure for imbalanced data classification task (i e , Anger/Happy emotion classification) 
Table TABREF14 show the results obtained for proposed s2sL approach in comparison to that of MLP for the tasks of Speech/Music and Neutral/Sad classification, by considering different proportions of training data  The values in Table TABREF14 are mean accuracies (in %) obtained by 5-fold cross validation  It can be observed from Table TABREF14 that for both tasks, s2sL method outperforms MLP, especially at low resource conditions  s2sL shows an absolute improvement in accuracy of INLINEFORM0 % and INLINEFORM1 % over MLP for Speech/Music and Neutral/Sad classification tasks, respectively, when INLINEFORM2 of the original training data is used in experiments 
Table TABREF14 show the results (in terms of INLINEFORM0 values) obtained for proposed s2sL approach in comparison to that of MLP for Anger/Happy classification (data imbalance problem)  Here, state-of-the-art methods i e , Eusboost [22] and MWMOTE [23] are also considered for comparison  It can be observed from Table TABREF14 that the s2sL method outperforms MLP, and also performs better than Eusboost and MWMOTE techniques on imbalanced data (around 3 % absolute improvement in INLINEFORM1 value for s2sL compared to MWMOTE, when INLINEFORM2 of the training data is considered)  In particular, at lower amounts of training data, s2sL outperforms all the other methods, illustrating its effectiveness even for low resourced data imbalance problems  s2sL method shows an absolute improvement of 6% ( INLINEFORM3 ) in INLINEFORM4 value over the second best ( INLINEFORM5 for MWMOTE), when only INLINEFORM6 of the training data is used 
Conclusions
In this paper, we introduced a novel s2s framework to effectively learn the class discriminative characteristics, even from low data resources  In this framework, more than one sample (here, two samples) are simultaneously considered to train the classifier  Further, this framework allows to generate multiple instances of the same test sample, by considering preselected reference samples, to achieve a more profound decision making  We illustrated the significance of our approach by providing the experimental results for two different tasks namely, speech/music discrimination and emotion classification  Further, we showed that the s2s framework can also handle the low resourced data imbalance problem 
References
[1] Hinton, G , Deng, L , Yu, D , Dahl, G  E , Mohamed, A  R , Jaitly, N , Senior, A , Vanhoucke, V , Nguyen, P , Sainath, T  N  & Kingsbury, B  (2012) Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups  IEEE Signal Processing Magazine, pp  82–97 
[2] Vinyals, O , Ravuri, S  V  & Povey, D  (2012) Revisiting recurrent neural networks for robust ASR  In Proc  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp  4085–4088 
[3] Lu, L , Zhang, X , Cho K  & Renals, S  (2015) A study of the recurrent neural network encoder-decoder for large vocabulary speech recognition  In Proc  INTERSPEECH, pp  3249–3253 
[4] Zhang, Y , Pezeshki, M , Brakel, P , Zhang, S , Bengio, C L Y  & Courville, A  (2017) Towards end-to-end speech recognition with deep convolutional neural networks  arXiv preprint, arXiv:1701 02720 
[5] Han, K , Yu, D  & Tashev, I  (2014) Speech emotion recognition using deep neural network and extreme learning machine  In Proc  Interspeech, pp  223–227 
[6] Trigeorgis, G , Ringeval, F , Brueckner, R , Marchi, E , Nicolaou, M  A , Schuller, B , & Zafeiriou, S  (2016) Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network  In Proc  ICASSP, pp  5200–5204) 
[7] Huang, Che-Wei, & Shrikanth S  Narayanan  (2016) Attention Assisted Discovery of Sub-Utterance Structure in Speech Emotion Recognition  In Proc  INTERSPEECH, pp  1387–1391 
[8] Zhang, Z , Ringeval, F , Han, J , Deng, J , Marchi, E  & Schuller, B  (2016) Facing realism in spontaneous emotion recognition from speech: Feature enhancement by autoencoder with LSTM neural networks  In Proc  INTERSPEECH, pp  3593–3597 
[9] Scheirer, E  D  & Slaney, M  (2003) Multi-feature speech/music discrimination system  U S  Patent 6,570,991 
[10] Pikrakis, A  & Theodoridis S  (2014) Speech-music discrimination: A deep learning perspective  In Proc  European Signal Processing Conference (EUSIPCO), pp  616–620 
[11] Choi, K , Fazekas, G  & Sandler, M  (2016) Automatic tagging using deep convolutional neural networks  arXiv preprint, arXiv:1606 00298 
[12] Zazo Candil, R , Sainath, T N , Simko, G  & Parada, C  (2016) Feature learning with raw-waveform CLDNNs for Voice Activity Detection  In Proc  Interspeech, pp  3668–3672 
[13] Tian, L , Moore, JD  & Lai, C  (2015) Emotion recognition in spontaneous and acted dialogues  In Proc  International Conference on Affective Computing and Intelligent Interaction (ACII), pp  698–704 
[14] Dumpala, S H , Chakraborty, R  & Kopparapu, S K  (2017) k-FFNN: A priori knowledge infused feed-forward neural networks  arXiv preprint, arXiv:1704 07055 
[15] Dumpala, S H , Chakraborty, R  & Kopparapu, S K  (2018) Knowledge-driven feed-forward neural network for audio affective content analysis  To appear in AAAI-2018 workshop on Affective Content Analysis 
[16] Deng, J , Zhang, Z , Marchi, E  & Schuller, B  (2013) Sparse autoencoder-based feature transfer learning for speech emotion recognition  In Proc  Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp  511–516 
[17] George Tzanetakis, Gtzan musicspeech  Availabe on-line at http://marsyas info/download/datasets 
[18] Burkhardt, F , Paeschke, A , Rolfes, M , Sendlmeier, W F  & Weiss, B  (2005) A database of german emotional speech  In Proc  Interspeech, pp  1517–1520 
[19] Hall, M , Frank, E , Holmes, G , Pfahringer, B , Reutemann, P  & Witten, I H  (2009) The WEKA data mining software: an update  ACM SIGKDD explorations newsletter, 11(1), pp  10–18 
[20] Eyben, F , Weninger, F , Gross, F  & Schuller, B  (2013) Recent developments in opensmile, the munich open-source multimedia feature extractor  In Proc  ACM international conference on Multimedia, pp  835–838 
[21] Maratea, A , Petrosino, A  & Manzo, M  (2014) Adjusted f-measure and kernel scaling for imbalanced data learning  Information Sciences 257:331–341 
[22] Galar, M , Fernandez, A , Barrenechea, E  & Herrera, F  (2013) Eusboost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling  Pattern Recognition 46(12):3460–3471 
[23] Barua, S , Islam, M  M , Yao, X  & Murase, K  (2014) Mwmote–majority weighted minority oversampling technique for imbalanced data set learning  IEEE Transactions on Knowledge and Data Engineering 26(2):405–425 
Context 2: Introduction
In June 2015, the operators of the online discussion site Reddit banned several communities under new anti-harassment rules  BIBREF0 used this opportunity to combine rich online data with computational methods to study a current question: Does eliminating these “echo chambers” diminish the amount of hate speech overall? Exciting opportunities like these, at the intersection of “thick” cultural and societal questions on the one hand, and the computational analysis of rich textual data on larger-than-human scales on the other, are becoming increasingly common 
Indeed, computational analysis is opening new possibilities for exploring challenging questions at the heart of some of the most pressing contemporary cultural and social issues  While a human reader is better equipped to make logical inferences, resolve ambiguities, and apply cultural knowledge than a computer, human time and attention are limited  Moreover, many patterns are not obvious in any specific context, but only stand out in the aggregate  For example, in a landmark study, BIBREF1 analyzed the authorship of The Federalist Papers using a statistical text analysis by focusing on style, based on the distribution of function words, rather than content  As another example, BIBREF2 studied what defines English haiku and showed how computational analysis and close reading can complement each other  Computational approaches are valuable precisely because they help us identify patterns that would not otherwise be discernible 
Yet these approaches are not a panacea  Examining thick social and cultural questions using computational text analysis carries significant challenges  For one, texts are culturally and socially situated  They reflect the ideas, values and beliefs of both their authors and their target audiences, and such subtleties of meaning and interpretation are difficult to incorporate in computational approaches  For another, many of the social and cultural concepts we seek to examine are highly contested — hate speech is just one such example  Choices regarding how to operationalize and analyze these concepts can raise serious concerns about conceptual validity and may lead to shallow or obvious conclusions, rather than findings that reflect the depth of the questions we seek to address 
These are just a small sample of the many opportunities and challenges faced in computational analyses of textual data  New possibilities and frustrating obstacles emerge at every stage of research, from identification of the research question to interpretation of the results  In this article, we take the reader through a typical research process that involves measuring social or cultural concepts using computational methods, discussing both the opportunities and complications that often arise  In the Reddit case, for example, hate speech is measured, however imperfectly, by the presence of particular words semi-automatically extracted from a machine learning algorithm  Operationalizations are never perfect translations, and are often refined over the course of an investigation, but they are crucial 
We begin our exploration with the identification of research questions, proceed through data selection, conceptualization, and operationalization, and end with analysis and the interpretation of results  The research process sounds more or less linear this way, but each of these phases overlaps, and in some instances turns back upon itself  The analysis phase, for example, often feeds back into the original research questions, which may continue to evolve for much of the project  At each stage, our discussion is critically informed by insights from the humanities and social sciences, fields that have focused on, and worked to tackle, the challenges of textual analysis—albeit at smaller scales—since their inception 
In describing our experiences with computational text analysis, we hope to achieve three primary goals  First, we aim to shed light on thorny issues not always at the forefront of discussions about computational text analysis methods  Second, we hope to provide a set of best practices for working with thick social and cultural concepts  Our guidance is based on our own experiences and is therefore inherently imperfect  Still, given our diversity of disciplinary backgrounds and research practices, we hope to capture a range of ideas and identify commonalities that will resonate for many  And this leads to our final goal: to help promote interdisciplinary collaborations  Interdisciplinary insights and partnerships are essential for realizing the full potential of any computational text analysis that involves social and cultural concepts, and the more we are able to bridge these divides, the more fruitful we believe our work will be 
Research questions
We typically start by identifying the questions we wish to explore  Can text analysis provide a new perspective on a “big question” that has been attracting interest for years? Or can we raise new questions that have only recently emerged, for example about social media? For social scientists working in computational analysis, the questions are often grounded in theory, asking: How can we explain what we observe? These questions are also influenced by the availability and accessibility of data sources  For example, the choice to work with data from a particular social media platform may be partly determined by the fact that it is freely available, and this will in turn shape the kinds of questions that can be asked  A key output of this phase are the concepts to measure, for example: influence; copying and reproduction; the creation of patterns of language use; hate speech  Computational analysis of text motivated by these questions is insight driven: we aim to describe a phenomenon or explain how it came about  For example, what can we learn about how and why hate speech is used or how this changes over time? Is hate speech one thing, or does it comprise multiple forms of expression? Is there a clear boundary between hate speech and other types of speech, and what features make it more or less ambiguous? In these cases, it is critical to communicate high-level patterns in terms that are recognizable 
This contrasts with much of the work in computational text analysis, which tends to focus on automating tasks that humans perform inefficiently  These tasks range from core linguistically motivated tasks that constitute the backbone of natural language processing, such as part-of-speech tagging and parsing, to filtering spam and detecting sentiment  Many tasks are motivated by applications, for example to automatically block online trolls  Success, then, is often measured by performance, and communicating why a certain prediction was made—for example, why a document was labeled as positive sentiment, or why a word was classified as a noun—is less important than the accuracy of the prediction itself  The approaches we use and what we mean by `success' are thus guided by our research questions 
Domain experts and fellow researchers can provide feedback on questions and help with dynamically revising them  For example, they may say “we already think we know that”, “that's too naïve”, “that doesn't reflect social reality” (negative); “two major camps in the field would give different answers to that question” (neutral); “we tried to look at that back in the 1960s, but we didn't have the technology” (positive); and “that sounds like something that people who made that archive would love”, “that's a really fundamental question” (very positive) 
Sometimes we also hope to connect to multiple disciplines  For example, while focusing on the humanistic concerns of an archive, we could also ask social questions such as “is this archive more about collaborative processes, culture-building or norm creation?” or “how well does this archive reflect the society in which it is embedded?" BIBREF3 used quantitative methods to tell a story about Darwin's intellectual development—an essential biographical question for a key figure in the history of science  At the same time, their methods connected Darwin's development to the changing landscape of Victorian scientific culture, allowing them to contrast Darwin's “foraging” in the scientific literature of his time to the ways in which that literature was itself produced  Finally, their methods provided a case study, and validation of technical approaches, for cognitive scientists who are interested in how people explore and exploit sources of knowledge 
Questions about potential “dual use” may also arise  Returning to our introductory example, BIBREF0 started with a deceptively simple question: if an internet platform eliminates forums for hate speech, does this impact hate speech in other forums? The research was motivated by the belief that a rising tide of online hate speech was (and is) making the internet increasingly unfriendly for disempowered groups, including minorities, women, and LBGTQ individuals  Yet the possibility of dual use troubled the researchers from the onset  Could the methodology be adopted to target the speech of groups like Black Lives Matter? Could it be adopted by repressive governments to minimize online dissent? While these concerns remained, they concluded that hypothetical dual use scenarios did not outweigh the tangible contribution this research could offer towards making the online environment more equal and just 
Data
The next step involves deciding on the data sources, collecting and compiling the dataset, and inspecting its metadata 
Data acquisition
Many scholars in the humanities and the social sciences work with sources that are not available in digital form, and indeed may never be digitized  Others work with both analogue and digitized materials, and the increasing digitization of archives has opened opportunities to study these archives in new ways  We can go to the canonical archive or open up something that nobody has studied before  For example, we might focus on major historical moments (French Revolution, post-Milosevic Serbia) or critical epochs (Britain entering the Victorian era, the transition from Latin to proto-Romance)  Or, we could look for records of how people conducted science, wrote and consumed literature, and worked out their philosophies 
A growing number of researchers work with born-digital sources or data  Born-digital data, e g , from social media, generally do not involve direct elicitation from participants and therefore enable unobtrusive measurements BIBREF5 , BIBREF6   In contrast, methods like surveys sometimes elicit altered responses from participants, who might adapt their responses to what they think is expected  Moreover, born-digital data is often massive, enabling large-scale studies of language and behavior in a variety of social contexts 
Still, many scholars in the social sciences and humanities work with multiple data sources  The variety of sources typically used means that more than one data collection method is often required  For example, a project examining coverage of a UK General Election, could draw data from traditional media, web archives, Twitter and Facebook, campaign manifestos, etc  and might combine textual analysis of these materials with surveys, laboratory experiments, or field observations offline  In contrast, many computational studies based on born-digital data have focused on one specific source, such as Twitter 
The use of born-digital data raises ethical concerns  Although early studies often treated privacy as a binary construct, many now acknowledge its complexity BIBREF7   Conversations on private matters can be posted online, visible for all, but social norms regarding what should be considered public information may differ from the data's explicit visibility settings  Often no informed consent has been obtained, raising concerns and challenges regarding publishing content and potentially harmful secondary uses BIBREF8 , BIBREF4  
Recently, concerns about potential harms stemming from secondary uses have led a number of digital service providers to restrict access to born-digital data  Facebook and Twitter, for example, have reduced or eliminated public access to their application programming interfaces (APIs) and expressed hesitation about allowing academic researchers to use data from their platforms to examine certain sensitive or controversial topics  Despite the seeming abundance of born-digital data, we therefore cannot take its availability for granted 
Working with data that someone else has acquired presents additional problems related to provenance and contextualisation  It may not always be possible to determine the criteria applied during the creation process  For example, why were certain newspapers digitized but not others, and what does this say about the collection? Similar questions arise with the use of born-digital data  For instance, when using the Internet Archive’s Wayback Machine to gather data from archived web pages, we need to consider what pages were captured, which are likely missing, and why 
We must often repurpose born-digital data (e g , Twitter was not designed to measure public opinion), but data biases may lead to spurious results and limit justification for generalization  In particular, data collected via black box APIs designed for commercial, not research, purposes are likely to introduce biases into the inferences we draw, and the closed nature of these APIs means we rarely know what biases are introduced, let alone how severely they might impact our research BIBREF10   These, however, are not new problems  Historians, for example, have always understood that their sources were produced within particular contexts and for particular purposes, which are not always apparent to us 
Non-representative data can still be useful for making comparisons within a sample  In the introductory example on hate speech BIBREF0 , the Reddit forums do not present a comprehensive or balanced picture of hate speech: the writing is almost exclusively in English, the targets of hate speech are mainly restricted (e g , to black people, or women), and the population of writers is shaped by Reddit's demographics, which skew towards young white men  These biases limit the generalizability of the findings, which cannot be extrapolated to other languages, other types of hate speech, and other demographic groups  However, because the findings are based on measurements on the same sort of hate speech and the same population of writers, as long as the collected data are representative of this specific population, these biases do not pose an intractable validity problem if claims are properly restricted 
The size of many newly available datasets is one of their most appealing characteristics  Bigger datasets often make statistics more robust  The size needed for a computational text analysis depends on the research goal: When it involves studying rare events, bigger datasets are needed  However, larger is not always better  Some very large archives are “secretly” collections of multiple and distinct processes that no in-field scholar would consider related  For example, Google Books is frequently used to study cultural patterns, but the over-representation of scientific articles in Google books can be problematic BIBREF11   Even very large born-digital datasets usually cover limited timespans compared to, e g , the Gutenberg archive of British novels 
This stage of the research also raises important questions about fairness  Are marginalized groups, for example, represented in the tweets we have collected? If not, what types of biases might result from analyses relying on those tweets?
Local experts and “informants” can help navigate the data  They can help understand the role an archive plays in the time and place  They might tell us: Is this the central archive, or a peripheral one? What makes it unusual? Or they might tell us how certain underrepresented communities use a social media platform and advise us on strategies for ensuring our data collection includes their perspectives 
However, when it is practically infeasible to navigate the data in this way—for instance, when we cannot determine what is missing from Twitter's Streaming API or what webpages are left out of the Internet Archive—we should be open about the limitations of our analyses, acknowledging the flaws in our data and drawing cautious and reasonable conclusions from them  In all cases, we should report the choices we have made when creating or re-using any dataset 
Compiling data
After identifying the data source(s), the next step is compiling the data  This step is fundamental: if the sources cannot support a convincing result, no result will be convincing  In many cases, this involves defining a “core" set of documents and a “comparison" set  We often have a specific set of documents in mind: an author's work, a particular journal, a time period  But if we want to say that this “core" set has some distinctive property, we need a “comparison" set  Expanding the collection beyond the documents that we would immediately think of has the beneficial effect of increasing our sample size  Having more sources increases the chance that we will notice something consistent across many individually varying contexts 
Comparing sets of documents can sometimes support causal inference, presented as a contrast between a treatment group and a control  In BIBREF0 , the treatment consisted of the text written in the two forums that were eventually closed by Reddit  However, identifying a control group required a considerable amount of time and effort  Reddit is a diverse platform, with a wide variety of interactional and linguistic styles; it would be pointless to compare hate speech forums against forums dedicated to, say, pictures of wrecked bicycles  Chandrasekharan et al  used a matching design, populating the control group with forums that were as similar as possible to the treatment group, but were not banned from Reddit  The goal is to estimate the counterfactual scenario: in this case, what would have happened had the site not taken action against these specific forums? An ideal control would make it possible to distinguish the effect of the treatment — closing the forums — from other idiosyncratic properties of texts that were treated 
We also look for categories of documents that might not be useful  We might remove documents that are meta-discourse, like introductions and notes, or documents that are in a language that is not the primary language of the collection, or duplicates when we are working with archived web pages  However, we need to carefully consider the potential consequences of information we remove  Does its removal alter the data, or the interpretation of the data, we are analyzing? Are we losing anything that might be valuable at a later stage?
Labels and metadata
Sometimes all we have is documents, but often we want to look at documents in the context of some additional information, or metadata  This additional information could tell us about the creation of documents (date, author, forum), or about the reception of documents (flagged as hate speech, helpful review)  Information about text segments can be extremely valuable, but it is also prone to errors, inconsistencies, bias, and missing information  Examining metadata is a good way to check a collection's balance and representativeness  Are sources disproportionately of one form? Is the collection missing a specific time window? This type of curation can be extremely time consuming as it may require expert labeling, but it often leads to the most compelling results  Sometimes metadata are also used as target labels to develop machine learning models  But using them as a “ground truth” requires caution  Labels sometimes mean something different than we expect  For example, a down vote for a social media post could indicate that the content is offensive, or that the voter simply disagreed with the expressed view 
Conceptualization
A core step in many analyses is translating social and cultural concepts (such as hate speech, rumor, or conversion) into measurable quantities  Before we can develop measurements for these concepts (the operationalization step, or the “implementation” step as denoted by BIBREF12 ), we need to define them  In the conceptualization phase we often start with questions such as: who are the domain experts, and how have they approached the topic? We are looking for a definition of the concept that is flexible enough to apply on our dataset, yet formal enough for computational research  For example, our introductory study on hate speech BIBREF0 used a statement on hate speech produced by the European Union Court of Human Rights  The goal was not to implement this definition directly in software but to use it as a reference point to anchor subsequent analyses 
If we want to move beyond the use of ad hoc definitions, it can be useful to distinguish between what political scientists Adcock and Collier call the “background concept” and the “systematized concept” BIBREF13   The background concept comprises the full and diverse set of meanings that might be associated with a particular term  This involves delving into theoretical, conceptual, and empirical studies to assess how a concept has been defined by other scholars and, most importantly, to determine which definition is most appropriate for the particular research question and the theoretical framework in which it is situated  That definition, in turn, represents the systematized concept: the formulation that is adopted for the study 
It is important to consider that for social and cultural concepts there is no absolute ground truth  There are often multiple valid definitions for a concept (the “background” concept in the terms of Adcock and Collier), and definitions might be contested over time  This may be uncomfortable for computer scientists, whose primary measure of success is often based on comparing a model's output against “ground truth” or a “gold standard”, e g , by comparing a sentiment classifier's output against manual annotations  However, the notion of ground truth is uncommon in the humanities and the social sciences and it is often taken too far in machine learning  BIBREF14 notes that in literary criticism and the digital humanities more broadly “interpretation, ambiguity, and argumentation are prized far above ground truth and definitive conclusions"  BIBREF15 draw attention to the different attitudes of literary scholars and computational linguists towards ambiguity, stating that “In Computational Linguistics [  ] ambiguity is almost uniformly treated as a problem to be solved; the focus is on disambiguation, with the assumption that one true, correct interpretation exists " The latter is probably true for tasks such as spam filtering, but in the social sciences and the humanities many relevant concepts are fundamentally unobservable, such as latent traits of political actors BIBREF16 or cultural fit in organizations BIBREF17 , leading to validation challenges  Moreover, when the ground truth comes from people, it may be influenced by ideological priors, priming, simple differences of opinion or perspective, and many other factors BIBREF18   We return to this issue in our discussions on validation and analysis 
Operationalization
In this phase we develop measures (or, “operationalizations”, or “indicators”) for the concepts of interest, a process called “operationalization”  Regardless of whether we are working with computers, the output produced coincides with Adcock and Collier's “scores”—the concrete translation and output of the systematized concept into numbers or labels BIBREF13   Choices made during this phase are always tied to the question “Are we measuring what we intend to measure?” Does our operationalization match our conceptual definition? To ensure validity we must recognize gaps between what is important and what is easy to measure  We first discuss modeling considerations  Next, we describe several frequently used computational approaches and their limitations and strengths 
Modeling considerations
The variables (both predictors and outcomes) are rarely simply binary or categorical  For example, a study on language use and age could focus on chronological age (instead of, e g , social age BIBREF19 )  However, even then, age can be modeled in different ways  Discretization can make the modeling easier and various NLP studies have modeled age as a categorical variable BIBREF20   But any discretization raises questions: How many categories? Where to place the boundaries? Fine distinctions might not always be meaningful for the analysis we are interested in, but categories that are too broad can threaten validity  Other interesting variables include time, space, and even the social network position of the author  It is often preferable to keep the variable in its most precise form  For example, BIBREF21 perform exploration in the context of hypothesis testing by using latitude and longitude coordinates — the original metadata attached to geotagged social media such as tweets — rather than aggregating into administrative units such as counties or cities  This is necessary when such administrative units are unlikely to be related to the target concept, as is the case in their analysis of dialect differences  Focusing on precise geographical coordinates also makes it possible to recognize fine-grained effects, such as language variation across the geography of a city 
Using a particular classification scheme means deciding which variations are visible, and which ones are hidden BIBREF22   We are looking for a categorization scheme for which it is feasible to collect a large enough labeled document collection (e g , to train supervised models), but which is also fine-grained enough for our purposes  Classification schemes rarely exhibit the ideal properties, i e , that they are consistent, their categories are mutually exclusive, and that the system is complete BIBREF22   Borderline cases are challenging, especially with social and cultural concepts, where the boundaries are often not clear-cut  The choice of scheme can also have ethical implications BIBREF22   For example, gender is usually represented as a binary variable in NLP and computational models tend to learn gender-stereotypical patterns  The operationalization of gender in NLP has been challenged only recently BIBREF23 , BIBREF24 , BIBREF25  
Supervised and unsupervised learning are the most common approaches to learning from data  With supervised learning, a model learns from labeled data (e g , social media messages labeled by sentiment) to infer (or predict) these labels from unlabeled texts  In contrast, unsupervised learning uses unlabeled data  Supervised approaches are especially suitable when we have a clear definition of the concept of interest and when labels are available (either annotated or native to the data)  Unsupervised approaches, such as topic models, are especially useful for exploration  In this setting, conceptualization and operationalization may occur simultaneously, with theory emerging from the data BIBREF26   Unsupervised approaches are also used when there is a clear way of measuring a concept, often based on strong assumptions  For example, BIBREF3 measure “surprise” in an analysis of Darwin's reading decisions based on the divergence between two probability distributions 
From an analysis perspective, the unit of text that we are labeling (or annotating, or coding), either automatic or manual, can sometimes be different than one's final unit of analysis  For example, if in a study on media frames in news stories, the theoretical framework and research question point toward frames at the story level (e g , what is the overall causal analysis of the news article?), the story must be the unit of analysis  Yet it is often difficult to validly and reliably code a single frame at the story level  Multiple perspectives are likely to sit side-by-side in a story  Thus, an article on income inequality might point to multiple causes, such as globalization, education, and tax policies  Coding at the sentence level would detect each of these causal explanations individually, but this information would need to be somehow aggregated to determine the overall story-level frame  Sometimes scholars solve this problem by only examining headlines and lead paragraphs, arguing that based on journalistic convention, the most important information can be found at the beginning of a story  However, this leads to a return to a shorter, less nuanced analysis 
From a computational perspective, the unit of text can also make a huge difference, especially when we are using bag-of-words models, where word order within a unit does not matter  Small segments, like tweets, sometimes do not have enough information to make their semantic context clear  In contrast, larger segments, like novels, have too much variation, making it difficult to train focused models  Finding a good segmentation sometimes means combining short documents and subdividing long documents  The word “document" can therefore be misleading  But it is so ingrained in the common NLP lexicon that we use it anyway in this article 
For insight-driven text analysis, it is often critical that high-level patterns can be communicated  Furthermore, interpretable models make it easier to find spurious features, to do error analysis, and to support interpretation of results  Some approaches are effective for prediction, but harder to interpret  The value we place on interpretability can therefore influence the approach we choose  There is an increasing interest in developing interpretable or transparent models in the NLP and machine learning communities 
Annotation
Many studies involve human coders  Sometimes the goal is to fully code the data, but in a computational analysis we often use the labels (or annotations) to train machine learning models to automatically recognize them, and to identify language patterns that are associated with these labels  For example, for a project analyzing rumors online BIBREF27 , conversation threads were annotated along different dimensions, including rumor versus non-rumor and stance towards a rumor 
The collection of annotation choices make up an annotation scheme (or “codebook”)  Existing schemes and annotations can be useful as starting points  Usually settling on an annotation scheme requires several iterations, in which the guidelines are updated and annotation examples are added  For example, a political scientist could use a mixed deductive-inductive strategy for developing a codebook  She starts by laying out a set of theory-driven deductive coding rules, which means that the broad principles of the coding rules are laid out without examining examples first  These are then tested (and possibly adjusted) based on a sample of the data  In line with Adcock and Collier's notion of “content validity” BIBREF13 , the goal is to assess whether the codebook adequately captures the systematized concept  By looking at the data themselves, she gains a better sense of whether some things have been left out of the coding rules and whether anything is superfluous, misleading, or confusing  Adjustments are made and the process is repeated, often with another researcher involved 
The final annotations can be collected using a crowdsourcing platform, a smaller number of highly-trained annotators, or a group of experts  Which type of annotator to use should be informed by the complexity and specificity of the concept  For more complex concepts, highly-trained or expert annotators tend to produce more reliable results  However, complex concepts can sometimes be broken down into micro-tasks that can be performed independently in parallel by crowdsourced annotators  Concepts from highly specialized domains may require expert annotators  In all cases, however, some training will be required, and the training phase should involve continual checks of inter-annotator agreement (i e  intercoder reliability) or checks against a gold standard (e g  quizzes in crowdsourcing platforms) 
We also need to decide how inter-annotator agreement will be measured and what an acceptable level of agreement would be  Krippendorff's alpha is frequently used in the social sciences, but the right measure depends on the type of data and task  For manual coding, we can continually check inter-annotator agreement and begin introducing checks of intra-annotator agreement, too  For most communication scholars using only manual content analysis, an acceptable rate of agreement is achieved when Krippendorf's alpha reaches 0 80 or above  When human-coded data are used to validate machine learning algorithms, the reliability of the human-coded data is even more important  Disagreement between annotators can signal weaknesses of the annotation scheme, or highlight the inherent ambiguity in what we are trying to measure  Disagreement itself can be meaningful and can be integrated in subsequent analyses BIBREF28 , BIBREF29  
Data pre-processing
Preparing the data can be a complex and time-consuming process, often involving working with partially or wholly unstructured data  The pre-processing steps have a big impact on the operationalizations, subsequent analyses and reproducibility efforts BIBREF30 , and they are usually tightly linked to what we intend to measure  Unfortunately, these steps tend to be underreported, but documenting the pre-processing choices made is essential and is analogous to recording the decisions taken during the production of a scholarly edition or protocols in biomedical research  Data may also vary enormously in quality, depending on how it has been generated  Many historians, for example, work with text produced from an analogue original using Optical Character Recognition (OCR)  Often, there will be limited information available regarding the accuracy of the OCR, and the degree of accuracy may even vary within a single corpus (e g  where digitized text has been produced over a period of years, and the software has gradually improved)  The first step, then, is to try to correct for common OCR errors  These will vary depending on the type of text, the date at which the `original' was produced, and the nature of the font and typesetting 
One step that almost everyone takes is to tokenize the original character sequence into the words and word-like units  Tokenization is a more subtle and more powerful process than people expect  It is often done using regular expressions or scripts that have been circulating within the NLP community  Tokenization heuristics, however, can be badly confused by emoticons, creative orthography (e g , U$A, sh!t), and missing whitespace  Multi-word terms are also challenging  Treating them as a single unit can dramatically alter the patterns in text  Many words that are individually ambiguous have clear, unmistakable meanings as terms, like “black hole" or “European Union"  However, deciding what constitutes a multi-word term is a difficult problem  In writing systems like Chinese, tokenization is a research problem in its own right 
Beyond tokenization, common steps include lowercasing, removing punctuation, stemming (removing suffixes), lemmatization (converting inflections to a base lemma), and normalization, which has never been clearly defined, but often includes grouping abbreviations like “U S A " and “USA", ordinals like “1st" and “first", and variant spellings like “noooooo"  The main goal of these steps is to improve the ratio of tokens (individual occurrences) to types (the distinct things in a corpus)  Each step requires making additional assumptions about which distinctions are relevant: is “apple” different from “Apple”? Is “burnt” different from “burned”? Is “cool" different from “coooool"? Sometimes these steps can actively hide useful patterns, like social meaning BIBREF32   Some of us therefore try do as little modification as possible 
From a multilingual perspective, English and Chinese have an unusually simple inflectional system, and so it is statistically reasonable to treat each inflection as a unique word type  Romance languages have considerably more inflections than English; many indigenous North American languages have still more  For these languages, unseen data is far more likely to include previously-unseen inflections, and therefore, dealing with inflections is more important  On the other hand, the resources for handling inflections vary greatly by language, with European languages dominating the attention of the computational linguistics community thus far 
We sometimes also remove words that are not relevant to our goals, for example by calculating vocabulary frequencies  We construct a “stoplist” of words that we are not interested in  If we are looking for semantic themes we might remove function words like determiners and prepositions  If we are looking for author-specific styles, we might remove all words except function words  Some words are generally meaningful but too frequent to be useful within a specific collection  We sometimes also remove very infrequent words  Their occurrences are too low for robust patterns and removing them helps reducing the vocabulary size 
The choice of processing steps can be guided by theory or knowledge about the domain as well as experimental investigation  When we have labels, predictive accuracy of a model is a way to assess the effect of the processing steps  In unsupervised settings, it is more challenging to understand the effects of different steps  Inferences drawn from unsupervised settings can be sensitive to pre-processing choices BIBREF33   Stemming has been found to provide little measurable benefits for topic modeling and can sometimes even be harmful BIBREF34   All in all, this again highlights the need to document these steps 
Finally, we can also mark up the data, e g , by identifying entities (people, places, organizations, etc ) or parts of speech  Although many NLP tools are available for such tasks, they are often challenged by linguistic variation, such as orthographic variation in historical texts BIBREF35 and social media BIBREF32   Moreover, the performance of NLP tools often drops when applying them outside the training domain, such as applying tools developed on newswire texts to texts written by younger authors BIBREF36   Problems (e g , disambiguation in named entity recognition) are sometimes resolved using considerable manual intervention  This combination of the automated and the manual, however, becomes more difficult as the scale of the data increases, and the `certainty' brought by the latter may have to be abandoned 
Dictionary-based approaches
Dictionaries are frequently used to code texts in content analyses BIBREF37   Dictionaries consist of one or more categories (i e  word lists)  Sometimes the output is simply the number of category occurrences (e g , positive sentiment), thus weighting words within a category equally  In some other cases, words are assigned continuous scores  The high transparency of dictionaries makes them sometimes more suitable than supervised machine learning models  However, dictionaries should only be used if the scores assigned to words match how the words are used in the data (see BIBREF38 for a detailed discussion on limitations)  There are many off-the-shelf dictionaries available (e g , LIWC BIBREF39 )  These are often well-validated, but applying them on a new domain may not be appropriate without additional validation  Corpus- or domain-specific dictionaries can overcome limitations of general-purpose dictionaries 
The dictionaries are often manually compiled, but increasingly they are constructed semi-automatically (e g , BIBREF40 )  When we semi-automatically create a word list, we use automation to identify an initial word list, and human insight to filter it  By automatically generating the initial words lists, words can be identified that human annotators might have difficulty intuiting  By manually filtering the lists, we use our theoretical understanding of the target concept to remove spurious features 
In the introduction study, SAGE BIBREF41 was used to obtain a list of words that distinguished the text in the treatment group (subreddits that were closed by Reddit) from text in the control group (similar subreddits that were not closed)  The researchers then returned to the hate speech definition provided by the European Court of Human Rights, and manually filtered the top SAGE words based on this definition  Not all identified words fitted the definition  The others included: the names of the subreddits themselves, names of related subreddits, community-specific jargon that was not directly related to hate speech, and terms such as IQ and welfare, which were frequently used in discourses of hate speech, but had significant other uses  The word lists provided the measurement instrument for their main result, which is that the use of hate speech throughout Reddit declined after the two treatment subreddits were closed 
Supervised models
Supervised learning is frequently used to scale up analyses  For example, BIBREF42 wanted to analyze the motivations of Movember campaign participants  By developing a classifier based on a small set of annotations, they were able to expand the analysis to over 90k participants 
The choice of supervised learning model is often guided by the task definition and the label types  For example, to identify stance towards rumors based on sequential annotations, an algorithm for learning from sequential BIBREF43 or time series data BIBREF44 could be used  The features (sometimes called variables or predictors) are used by the model to make the predictions  They may vary from content-based features such as single words, sequences of words, or information about their syntactic structure, to meta-information such as user or network information  Deciding on the features requires experimentation and expert insight and is often called feature engineering  For insight-driven analysis, we are often interested in why a prediction has been made and features that can be interpreted by humans may be preferred  Recent neural network approaches often use simple features as input (such as word embeddings or character sequences), which requires less feature engineering but make interpretation more difficult 
Supervised models are powerful, but they can latch on to spurious features of the dataset  This is particularly true for datasets that are not well-balanced, and for annotations that are noisy  In our introductory example on hate speech in Reddit BIBREF0 , the annotations are automatically derived from the forum in which each post appears, and indeed, many of the posts in the forums (subreddits) that were banned by Reddit would be perceived by many as hate speech  But even in banned subreddits, not all of the content is hate speech (e g , some of the top features were self-referential like the name of the subreddit) but a classifier would learn a high weight for these features 
Even when expert annotations are available on the level of individual posts, spurious features may remain  BIBREF45 produced expert annotations of hate speech on Twitter  They found that one of the strongest features for sexism is the name of an Australian TV show, because people like to post sexist comments about the contestants  If we are trying to make claims about what inhibits or encourages hate speech, we would not want those claims to be tied to the TV show's popularity  Such problems are inevitable when datasets are not well-balanced over time, across genres, topics, etc  Especially with social media data, we lack a clear and objective definition of `balance' at this time 
The risk of supervised models latching on to spurious features reinforces the need for interpretability  Although the development of supervised models is usually performance driven, placing more emphasis on interpretability could increase the adoption of these models in insight-driven analyses  One way would be to only use models that are already somewhat interpretable, for example models that use a small number of human-interpretable features  Rather than imposing such restrictions, there is also work on generating post-hoc explanations for individual predictions (e g , BIBREF46 ), even when the underlying model itself is very complex 
Topic modeling
Topic models (e g , LDA BIBREF47 ) are usually unsupervised and therefore less biased towards human-defined categories  They are especially suited for insight-driven analysis, because they are constrained in ways that make their output interpretable  Although there is no guarantee that a “topic” will correspond to a recognizable theme or event or discourse, they often do so in ways that other methods do not  Their easy applicability without supervision and ready interpretability make topic models good for exploration  Topic models are less successful for many performance-driven applications  Raw word features are almost always better than topics for search and document classification  LSTMs and other neural network models are better as language models  Continuous word embeddings have more expressive power to represent fine-grained semantic similarities between words 
A topic model provides a different perspective on a collection  It creates a set of probability distributions over the vocabulary of the collection, which, when combined together in different proportions, best match the content of the collection  We can sort the words in each of these distributions in descending order by probability, take some arbitrary number of most-probable words, and get a sense of what (if anything) the topic is “about”  Each of the text segments also has its own distribution over the topics, and we can sort these segments by their probability within a given topic to get a sense of how that topic is used 
One of the most common questions about topic models is how many topics to use, usually with the implicit assumption that there is a “right” number that is inherent in the collection  We prefer to think of this parameter as more like the scale of a map or the magnification of a microscope  The “right” number is determined by the needs of the user, not by the collection  If the analyst is looking for a broad overview, a relatively small number of topics may be best  If the analyst is looking for fine-grained phenomena, a larger number is better 
After fitting the model, it may be necessary to circle back to an earlier phase  Topic models find consistent patterns  When authors repeatedly use a particular theme or discourse, that repetition creates a consistent pattern  But other factors can also create similar patterns, which look as good to the algorithm  We might notice a topic that has highest probability on French stopwords, indicating that we need to do a better job of filtering by language  We might notice a topic of word fragments, such as “ing”, “tion”, “inter”, indicating that we are not handling end-of-line hyphenation correctly  We may need to add to our stoplist or change how we curate multi-word terms 
Validation
The output of our measurement procedures (in the social sciences often called the “scores”) must now be assessed in terms of their reliability and validity with regard to the (systemized) concept  Reliability aims to capture repeatability, i e  the extent to which a given tool provides consistent results 
Validity assesses the extent to which a given measurement tool measures what it is supposed to measure  In NLP and machine learning, most models are primarily evaluated by comparing the machine-generated labels against an annotated sample  This approach presumes that the human output is the “gold standard" against which performance should be tested  In contrast, when the reliability is measured based on the output of different annotators, no coder is taken as the standard and the likelihood of coders reaching agreement by chance (rather than because they are “correct") is factored into the resulting statistic  Comparing against a “gold standard” suggests that the threshold for human inter- and intra-coder reliability should be particularly high 
Accuracy, as well as other measures such as precision, recall and F-score, are sometimes presented as a measure of validity, but if we do not have a genuinely objective determination of what something is supposed measure—as is often the case in text analysis—then accuracy is perhaps a better indication of reliability than of validity  In that case, validity needs to be assessed based on other techniques like those we discuss later in this section  It is also worth asking what level of accuracy is sufficient for our analysis and to what extent there may be an upper bound, especially when the labels are native to the data or when the notion of a “gold standard” is not appropriate 
For some in the humanities, validation takes the form of close reading, not designed to confirm whether the model output is correct, but to present what BIBREF48 refers to as a form of “further discovery in two directions”  Model outputs tell us something about the texts, while a close reading of the texts alongside those outputs tells us something about the models that can be used for more effective model building  Applying this circular, iterative process to 450 18th-century novels written in three languages, Piper was able to uncover a new form of “conversional novel” that was not previously captured in “literary history's received critical categories” BIBREF48  
Along similar lines, we can subject both the machine-generated output and the human annotations to another round of content validation  That is, take a stratified random sample, selecting observations from the full range of scores, and ask: Do these make sense in light of the systematized concept? If not, what seems to be missing? Or is something extraneous being captured? This is primarily a qualitative process that requires returning to theory and interrogating the systematized concept, indicators, and scores together  This type of validation is rarely done in NLP, but it is especially important when it is difficult to assess what drives a given machine learning model  If there is a mismatch between the scores and systematized concept at this stage, the codebook may need to be adjusted, human coders retrained, more training data prepared, algorithms adjusted, or in some instances, even a new analytical method adopted 
Other types of validation are also possible, such as comparing with other approaches that aim to capture the same concept, or comparing the output with external measures (e g , public opinion polls, the occurrence of future events)  We can also go beyond only evaluating the labels (or point estimates)  BIBREF16 used human judgments to not only assess the positional estimates from a scaling method of latent political traits but also to assess uncertainty intervals  Using different types of validation can increase our confidence in the approach, especially when there is no clear notion of ground truth 
Besides focusing on rather abstract evaluation measures, we could also assess the models in task-based settings using human experts  Furthermore, for insight-driven analyses, it can be more useful to focus on improving explanatory power than making small improvements in predictive performance 
Analysis
In this phase, we use our models to explore or answer our research questions  For example, given a topic model we can look at the connection between topics and metadata elements  Tags such as “hate speech" or metadata information imply a certain way of organizing the collection  Computational models provide another organization, which may differ in ways that provide more insight into how these categories manifest themselves, or fail to do so 
Moreover, when using a supervised approach, the “errors”, i e  disagreement between the system output and human-provided labels, can point towards interesting cases for closer analysis and help us reflect on our conceptualizations  In the words of BIBREF2 , they can be “opportunities for interpretation”  Other types of “failures” can be insightful as well  Sometimes there is a “dog that didn't bark” BIBREF49 –i e , something that everyone thinks we should have found, but we did not  Or, sometimes the failures are telling us about the existence of something in the data that nobody noticed, or thought important, until then (e g , the large number of travel journals in Darwin's reading lists) 
Computational text analysis is not a replacement for but rather an addition to the approaches one can take to analyze social and cultural phenomena using textual data  By moving back and forth between large-scale computational analyses and small-scale qualitative analyses, we can combine their strengths so that we can identify large-scale and long-term trends, but also tell individual stories  For example, the Reddit study on hate speech BIBREF0 raised various follow-up questions: Can we distinguish hate speech from people talking about hate speech? Did people find new ways to express hate speech? If so, did the total amount of online hate speech decrease after all? As possible next steps, a qualitative discourse analyst might examine a smaller corpus to investigate whether commenters were indeed expressing hate speech in new ways; a specialist in interview methodologies might reach out to commenters to better understand the role of online hate speech in their lives  Computational text analysis represents a step towards better understanding social and cultural phenomena, and it is in many cases better suited towards opening questions rather than closing them 
Conclusion
Insight-driven computational analysis of text is becoming increasingly common  It not only helps us see more broadly, it helps us see subtle patterns more clearly and allows us to explore radical new questions about culture and society  In this article we have consolidated our experiences, as scholars from very different disciplines, in analyzing text as social and cultural data and described how the research process often unfolds  Each of the steps in the process is time-consuming and labor-intensive  Each presents challenges  And especially when working across disciplines, the research often involves a fair amount of discussion—even negotiation—about what means of operationalization and approaches to analysis are appropriate and feasible  And yet, with a bit of perseverance and mutual understanding, conceptually sound and meaningful work results so that we can truly make use of the exciting opportunities rich textual data offers 
Acknowledgements
This work was supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1  Dong Nguyen is supported with an Alan Turing Institute Fellowship (TU/A/000006)  Maria Liakata is a Turing fellow at 40%  We would also like to thank the participants of the “Bridging disciplines in analysing text as social and cultural data” workshop held at the Turing Institute (2017) for insightful discussions  The workshop was funded by a Turing Institute seed funding award to Nguyen and Liakata 

ground_truth: MLP

Eusboost, MWMOTE

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What background do they have?
Context 4: Introduction
Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine reading comprehension  Table TABREF3 gives concrete examples: for the Named Entity Recognition (NER) task BIBREF2, BIBREF3, most tokens are backgrounds with tagging class $O$  Specifically, the number of tokens tagging class $O$ is 5 times as many as those with entity labels for the CoNLL03 dataset and 8 times for the OntoNotes5 0 dataset; Data-imbalanced issue is more severe for MRC tasks BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8 with the value of negative-positive ratio being 50-200 
Data imbalance results in the following two issues: (1) the training-test discrepancy: Without balancing the labels, the learning process tends to converge to a point that strongly biases towards class with the majority label  This actually creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function while at test time, F1 score concerns more about positive examples; (2) the overwhelming effect of easy-negative examples  As pointed out by meng2019dsreg, significantly large number of negative examples also means that the number of easy-negative example is large  The huge number of easy examples tends to overwhelm the training, making the model not sufficiently learned to distinguish between positive examples and hard-negative examples  The cross-entropy objective (CE for short) or maximum likelihood (MLE) objective, which is widely adopted as the training objective for data-imbalanced NLP tasks BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, handles neither of the issues 
To handle the first issue, we propose to replace CE or MLE with losses based on the Sørensen–Dice coefficient BIBREF0 or Tversky index BIBREF1  The Sørensen–Dice coefficient, dice loss for short, is the harmonic mean of precision and recall  It attaches equal importance to false positives (FPs) and false negatives (FNs) and is thus more immune to data-imbalanced datasets  Tversky index extends dice loss by using a weight that trades precision and recall, which can be thought as the approximation of the $F_{\beta }$ score, and thus comes with more flexibility  Therefore, We use dice loss or Tversky index to replace CE loss to address the first issue 
Only using dice loss or Tversky index is not enough since they are unable to address the dominating influence of easy-negative examples  This is intrinsically because dice loss is actually a hard version of the F1 score  Taking the binary classification task as an example, at test time, an example will be classified as negative as long as its probability is smaller than 0 5, but training will push the value to 0 as much as possible  This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily  Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones  Inspired by the idea of focal loss BIBREF16 in computer vision, we propose a dynamic weight adjusting strategy, which associates each training example with a weight in proportion to $(1-p)$, and this weight dynamically changes as training proceeds  This strategy helps to deemphasize confident examples during training as their $p$ approaches the value of 1, makes the model attentive to hard-negative examples, and thus alleviates the dominating effect of easy-negative examples 
Combing both strategies, we observe significant performance boosts on a wide range of data imbalanced NLP tasks  Notably, we are able to achieve SOTA results on CTB5 (97 92, +1 86), CTB6 (96 57, +1 80) and UD1 4 (96 98, +2 19) for the POS task; SOTA results on CoNLL03 (93 33, +0 29), OntoNotes5 0 (92 07, +0 96)), MSRA 96 72(+0 97) and OntoNotes4 0 (84 47,+2 36) for the NER task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification 
The rest of this paper is organized as follows: related work is presented in Section 2  We describe different training objectives in Section 3  Experimental results are presented in Section 4  We perform ablation studies in Section 5, followed by a brief conclusion in Section 6 
Related Work ::: Data Resample
The idea of weighting training examples has a long history  Importance sampling BIBREF17 assigns weights to different samples and changes the data distribution  Boosting algorithms such as AdaBoost BIBREF18 select harder examples to train subsequent classifiers  Similarly, hard example mining BIBREF19 downsamples the majority class and exploits the most difficult examples  Oversampling BIBREF20, BIBREF21 is used to balance the data distribution  Another line of data resampling is to dynamically control the weights of examples as training proceeds  For example, focal loss BIBREF16 used a soft weighting scheme that emphasizes harder examples during training  In self-paced learning BIBREF22, example weights are obtained through optimizing the weighted training loss which encourages learning easier examples first  At each training step, self-paced learning algorithm optimizes model parameters and example weights jointly  Other works BIBREF23, BIBREF24 adjusted the weights of different training examples based on training loss  Besides, recent work BIBREF25, BIBREF26 proposed to learn a separate network to predict sample weights 
Related Work ::: Data Imbalance Issue in Object Detection
The background-object label imbalance issue is severe and thus well studied in the field of object detection BIBREF27, BIBREF28, BIBREF29, BIBREF30, BIBREF31  The idea of hard negative mining (HNM) BIBREF30 has gained much attention recently  shrivastava2016ohem proposed the online hard example mining (OHEM) algorithm in an iterative manner that makes training progressively more difficult, and pushes the model to learn better  ssd2016liu sorted all of the negative samples based on the confidence loss and picking the training examples with the negative-positive ratio at 3:1  pang2019rcnn proposed a novel method called IoU-balanced sampling and aploss2019chen designed a ranking model to replace the conventional classification task with a average-precision loss to alleviate the class imbalance issue  The efforts made on object detection have greatly inspired us to solve the data imbalance issue in NLP 
Losses ::: Notation
For illustration purposes, we use the binary classification task to demonstrate how different losses work  The mechanism can be easily extended to multi-class classification 
Let $\lbrace x_i\rbrace $ denote a set of instances  Each $x_i$ is associated with a golden label vector $y_i = [y_{i0},y_{i1} ]$, where $y_{i1}\in \lbrace 0,1\rbrace $ and $y_{i0}\in \lbrace 0,1\rbrace $ respectively denote the positive and negative classes, and thus $y_i$ can be either $[0,1]$ or $[0,1]$  Let $p_i = [p_{i0},p_{i1} ]$ denote the probability vector, and $p_{i1}$ and $p_{i0}$ respectively denote the probability that a model assigns the positive and negative label to $x_i$ 
Losses ::: Cross Entropy Loss
The vanilla cross entropy (CE) loss is given by:
As can be seen from Eq DISPLAY_FORM8, each $x_i$ contributes equally to the final objective  Two strategies are normally used to address the the case where we wish that not all $x_i$ are treated equal: associating different classes with different weighting factor $\alpha $ or resampling the datasets  For the former, Eq DISPLAY_FORM8 is adjusted as follows:
where $\alpha _i\in [0,1]$ may be set by the inverse class frequency or treated as a hyperparameter to set by cross validation  In this work, we use $\lg (\frac{n-n_t}{n_t}+K)$ to calculate the coefficient $\alpha $, where $n_t$ is the number of samples with class $t$ and $n$ is the total number of samples in the training set  $K$ is a hyperparameter to tune  The data resampling strategy constructs a new dataset by sampling training examples from the original dataset based on human-designed criteria, e g , extract equal training samples from each class  Both strategies are equivalent to changing the data distribution and thus are of the same nature  Empirically, these two methods are not widely used due to the trickiness of selecting $\alpha $ especially for multi-class classification tasks and that inappropriate selection can easily bias towards rare classes BIBREF32 
Losses ::: Dice coefficient and Tversky index
Sørensen–Dice coefficient BIBREF0, BIBREF33, dice coefficient (DSC) for short, is a F1-oriented statistic used to gauge the similarity of two sets  Given two sets $A$ and $B$, the dice coefficient between them is given as follows:
In our case, $A$ is the set that contains of all positive examples predicted by a specific model, and $B$ is the set of all golden positive examples in the dataset  When applied to boolean data with the definition of true positive (TP), false positive (FP), and false negative (FN), it can be then written as follows:
For an individual example $x_i$, its corresponding DSC loss is given as follows:
As can be seen, for a negative example with $y_{i1}=0$, it does not contribute to the objective  For smoothing purposes, it is common to add a $\gamma $ factor to both the nominator and the denominator, making the form to be as follows:
As can be seen, negative examples, with $y_{i1}$ being 0 and DSC being $\frac{\gamma }{ p_{i1}+\gamma }$, also contribute to the training  Additionally, milletari2016v proposed to change the denominator to the square form for faster convergence, which leads to the following dice loss (DL):
Another version of DL is to directly compute set-level dice coefficient instead of the sum of individual dice coefficient  We choose the latter due to ease of optimization 
Tversky index (TI), which can be thought as the approximation of the $F_{\beta }$ score, extends dice coefficient to a more general case  Given two sets $A$ and $B$, tversky index is computed as follows:
Tversky index offers the flexibility in controlling the tradeoff between false-negatives and false-positives  It degenerates to DSC if $\alpha =\beta =0 5$  The Tversky loss (TL) for the training set $\lbrace x_i,y_i\rbrace $ is thus as follows:
Losses ::: Self-adusting Dice Loss
Consider a simple case where the dataset consists of only one example $x_i$, which is classified as positive as long as $p_{i1}$ is larger than 0 5  The computation of $F1$ score is actually as follows:
Comparing Eq DISPLAY_FORM14 with Eq DISPLAY_FORM22, we can see that Eq DISPLAY_FORM14 is actually a soft form of $F1$, using a continuous $p$ rather than the binary $\mathbb {I}( p_{i1}>0 5)$  This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily  Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones, which has a huge negative effect on the final F1 performance 
To address this issue, we propose to multiply the soft probability $p$ with a decaying factor $(1-p)$, changing Eq DISPLAY_FORM22 to the following form:
One can think $(1-p_{i1})$ as a weight associated with each example, which changes as training proceeds  The intuition of changing $p_{i1}$ to $(1-p_{i1}) p_{i1}$ is to push down the weight of easy examples  For easy examples whose probability are approaching 0 or 1, $(1-p_{i1}) p_{i1}$ makes the model attach significantly less focus to them  Figure FIGREF23 gives gives an explanation from the perspective in derivative: the derivative of $\frac{(1-p)p}{1+(1-p)p}$ with respect to $p$ approaches 0 immediately after $p$ approaches 0, which means the model attends less to examples once they are correctly classified 
A close look at Eq DISPLAY_FORM14 reveals that it actually mimics the idea of focal loss (FL for short) BIBREF16 for object detection in vision  Focal loss was proposed for one-stage object detector to handle foreground-background tradeoff encountered during training  It down-weights the loss assigned to well-classified examples by adding a $(1-p)^{\beta }$ factor, leading the final loss to be $(1-p)^{\beta }\log p$ 
In Table TABREF18, we show the losses used in our experiments, which is described in the next section 
Experiments
We evaluate the proposed method on four NLP tasks: part-of-speech tagging, named entity recognition, machine reading comprehension and paraphrase identification  Baselines in our experiments are optimized by using the standard cross-entropy training objective 
Experiments ::: Part-of-Speech Tagging
Part-of-speech tagging (POS) is the task of assigning a label (e g , noun, verb, adjective) to each word in a given text  In this paper, we choose BERT as the backbone and conduct experiments on three Chinese POS datasets  We report the span-level micro-averaged precision, recall and F1 for evaluation  Hyperparameters are tuned on the corresponding development set of each dataset 
Experiments ::: Part-of-Speech Tagging ::: Datasets
We conduct experiments on the widely used Chinese Treebank 5 0, 6 0 as well as UD1 4 
CTB5 is a Chinese dataset for tagging and parsing, which contains 507,222 words, 824,983 characters and 18,782 sentences extracted from newswire sources 
CTB6 is an extension of CTB5, containing 781,351 words, 1,285,149 characters and 28,295 sentences 
UD is the abbreviation of Universal Dependencies, which is a framework for consistent annotation of grammar (parts of speech, morphological features, and syntactic dependencies) across different human languages  In this work, we use UD1 4 for Chinese POS tagging 
Experiments ::: Part-of-Speech Tagging ::: Baselines
We use the following baselines:
Joint-POS: shao2017character jointly learns Chinese word segmentation and POS 
Lattice-LSTM: lattice2018zhang constructs a word-character lattice 
Bert-Tagger: devlin2018bert treats part-of-speech as a tagging task

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use pretrained embeddings?
Context 3: Introduction
Rendering natural language descriptions from structured data is required in a wide variety of commercial applications such as generating descriptions of products, hotels, furniture, etc , from a corresponding table of facts about the entity  Such a table typically contains {field, value} pairs where the field is a property of the entity (e g , color) and the value is a set of possible assignments to this property (e g , color = red)  Another example of this is the recently introduced task of generating one line biography descriptions from a given Wikipedia infobox BIBREF0   The Wikipedia infobox serves as a table of facts about a person and the first sentence from the corresponding article serves as a one line description of the person  Figure FIGREF2 illustrates an example input infobox which contains fields such as Born, Residence, Nationality, Fields, Institutions and Alma Mater  Each field further contains some words (e g , particle physics, many-body theory, etc )  The corresponding description is coherent with the information contained in the infobox 
Note that the number of fields in the infobox and the ordering of the fields within the infobox varies from person to person  Given the large size (700K examples) and heterogeneous nature of the dataset which contains biographies of people from different backgrounds (sports, politics, arts, etc ), it is hard to come up with simple rule-based templates for generating natural language descriptions from infoboxes, thereby making a case for data-driven models  Based on the recent success of data-driven neural models for various other NLG tasks BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , one simple choice is to treat the infobox as a sequence of {field, value} pairs and use a standard seq2seq model for this task  However, such a model is too generic and does not exploit the specific characteristics of this task as explained below  First, note that while generating such descriptions from structured data, a human keeps track of information at two levels  Specifically, at a macro level, she would first decide which field to mention next and then at a micro level decide which of the values in the field needs to be mentioned next  For example, she first decides that at the current step, the field occupation needs attention and then decides which is the next appropriate occupation to attend to from the set of occupations (actor, director, producer, etc )  To enable this, we use a bifocal attention mechanism which computes an attention over fields at a macro level and over values at a micro level  We then fuse these attention weights such that the attention weight for a field also influences the attention over the values within it  Finally, we feed a fused context vector to the decoder which contains both field level and word level information  Note that such two-level attention mechanisms BIBREF6 , BIBREF7 , BIBREF8 have been used in the context of unstructured data (as opposed to structured data in our case), where at a macro level one needs to pay attention to sentences and at a micro level to words in the sentences 
Next, we observe that while rendering the output, once the model pays attention to a field (say, occupation) it needs to stay on this field for a few timesteps (till all the occupations are produced in the output)  We refer to this as the stay on behavior  Further, we note that once the tokens of a field are referred to, they are usually not referred to later  For example, once all the occupations have been listed in the output we will never visit the occupation field again because there is nothing left to say about it  We refer to this as the never look back behavior  To model the stay on behaviour, we introduce a forget (or remember) gate which acts as a signal to decide when to forget the current field (or equivalently to decide till when to remember the current field)  To model the never look back behaviour we introduce a gated orthogonalization mechanism which ensures that once a field is forgotten, subsequent field context vectors fed to the decoder are orthogonal to (or different from) the previous field context vectors 
We experiment with the WikiBio dataset BIBREF0 which contains around 700K {infobox, description} pairs and has a vocabulary of around 400K words  We show that the proposed model gives a relative improvement of 21% and 20% as compared to current state of the art models BIBREF0 , BIBREF9 on this dataset  The proposed model also gives a relative improvement of 10% as compared to the basic seq2seq model  Further, we introduce new datasets for French and German on the same lines as the English WikiBio dataset  Even on these two datasets, our model outperforms the state of the art methods mentioned above 
Related work
Natural Language Generation has always been of interest to the research community and has received a lot of attention in the past  The approaches for NLG range from (i) rule based approaches (e g , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ) (ii) modular statistical approaches which divide the process into three phases (planning, selection and surface realization) and use data driven approaches for one or more of these phases BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 (iii) hybrid approaches which rely on a combination of handcrafted rules and corpus statistics BIBREF20 , BIBREF21 , BIBREF22 and (iv) the more recent neural network based models BIBREF1  
Neural models for NLG have been proposed in the context of various tasks such as machine translation BIBREF1 , document summarization BIBREF2 , BIBREF4 , paraphrase generation BIBREF23 , image captioning BIBREF24 , video summarization BIBREF25 , query based document summarization BIBREF5 and so on  Most of these models are data hungry and are trained on large amounts of data  On the other hand, NLG from structured data has largely been studied in the context of small datasets such as WeatherGov BIBREF26 , RoboCup BIBREF27 , NFL Recaps BIBREF15 , Prodigy-Meteo BIBREF28 and TUNA Challenge BIBREF29   Recently weather16 proposed RNN/LSTM based neural encoder-decoder models with attention for WeatherGov and RoboCup datasets 
Unlike the datasets mentioned above, the biography dataset introduced by lebret2016neural is larger (700K {table, descriptions} pairs) and has a much larger vocabulary (400K words as opposed to around 350 or fewer words in the above datasets)  Further, unlike the feed-forward neural network based model proposed by BIBREF0 we use a sequence to sequence model and introduce components to address the peculiar characteristics of the task  Specifically, we introduce neural components to address the need for attention at two levels and to address the stay on and never look back behaviour required by the decoder  KiddonZC16 have explored the use of checklists to track previously visited ingredients while generating recipes from ingredients  Note that two-level attention mechanisms have also been used in the context of summarization BIBREF6 , document classification BIBREF7 , dialog systems BIBREF8 , etc  However, these works deal with unstructured data (sentences at the higher level and words at a lower level) as opposed to structured data in our case 
Proposed model
As input we are given an infobox INLINEFORM0 , which is a set of pairs INLINEFORM1 where INLINEFORM2 corresponds to field names and INLINEFORM3 is the sequence of corresponding values and INLINEFORM4 is the total number of fields in INLINEFORM5   For example, INLINEFORM6 could be one such pair in this set  Given such an input, the task is to generate a description INLINEFORM7 containing INLINEFORM8 words  A simple solution is to treat the infobox as a sequence of fields followed by the values corresponding to the field in the order of their appearance in the infobox  For example, the infobox could be flattened to produce the following input sequence (the words in bold are field names which act as delimiters)
[Name] John Doe [Birth_Date] 19 March 1981 [Nationality] Indian      
The problem can then be cast as a seq2seq generation problem and can be modeled using a standard neural architecture comprising of three components (i) an input encoder (using GRU/LSTM cells), (ii) an attention mechanism to attend to important values in the input sequence at each time step and (iii) a decoder to decode the output one word at a time (again, using GRU/LSTM cells)  However, this standard model is too generic and does not exploit the specific characteristics of this task  We propose additional components, viz , (i) a fused bifocal attention mechanism which operates on fields (macro) and values (micro) and (ii) a gated orthogonalization mechanism to model stay on and never look back behavior 
Fused Bifocal Attention Mechanism
Intuitively, when a human writes a description from a table she keeps track of information at two levels  At the macro level, it is important to decide which is the appropriate field to attend to next and at a micro level (i e , within a field) it is important to know which values to attend to next  To capture this behavior, we use a bifocal attention mechanism as described below 
Macro Attention: Consider the INLINEFORM0 -th field INLINEFORM1 which has values INLINEFORM2   Let INLINEFORM3 be the representation of this field in the infobox  This representation can either be (i) the word embedding of the field name or (ii) some function INLINEFORM4 of the values in the field or (iii) a concatenation of (i) and (ii)  The function INLINEFORM5 could simply be the sum or average of the embeddings of the values in the field  Alternately, this function could be a GRU (or LSTM) which treats these values within a field as a sequence and computes the field representation as the final representation of this sequence (i e , the representation of the last time-step)  We found that bidirectional GRU is a better choice for INLINEFORM6 and concatenating the embedding of the field name with this GRU representation works best  Further, using a bidirectional GRU cell to take contextual information from neighboring fields also helps (these are the orange colored cells in the top-left block in Figure FIGREF3 with macro attention)  Given these representations INLINEFORM7 for all the INLINEFORM8 fields we compute an attention over the fields (macro level)  DISPLAYFORM0
where INLINEFORM0 is the state of the decoder at time step INLINEFORM1   INLINEFORM2 and INLINEFORM3 are parameters, INLINEFORM4 is the total number of fields in the input, INLINEFORM5 is the macro (field level) context vector at the INLINEFORM6 -th time step of the decoder 
Micro Attention: Let INLINEFORM0 be the representation of the INLINEFORM1 -th value in a given field  This representation could again either be (i) simply the embedding of this value (ii) or a contextual representation computed using a function INLINEFORM2 which also considers the other values in the field  For example, if INLINEFORM3 are the values in a field then these values can be treated as a sequence and the representation of the INLINEFORM4 -th value can be computed using a bidirectional GRU over this sequence  Once again, we found that using a bi-GRU works better then simply using the embedding of the value  Once we have such a representation computed for all values across all the fields, we compute the attention over these values (micro level) as shown below : DISPLAYFORM0
where INLINEFORM0 is the state of the decoder at time step INLINEFORM1   INLINEFORM2 and INLINEFORM3 are parameters, INLINEFORM4 is the total number of values across all the fields 
Fused Attention: Intuitively, the attention weights assigned to a field should have an influence on all the values belonging to the particular field  To ensure this, we reweigh the micro level attention weights based on the corresponding macro level attention weights  In other words, we fuse the attention weights at the two levels as: DISPLAYFORM0
where INLINEFORM0 is the field corresponding to the INLINEFORM1 -th value, INLINEFORM2 is the macro level context vector 
Gated Orthogonalization for Modeling Stay-On and Never Look Back behaviour
We now describe a series of choices made to model stay-on and never look back behavior  We first begin with the stay-on property which essentially implies that if we have paid attention to the field INLINEFORM0 at timestep INLINEFORM1 then we are likely to pay attention to the same field for a few more time steps  For example, if we are focusing on the occupation field at this timestep then we are likely to focus on it for the next few timesteps till all relevant values in this field have been included in the generated description  In other words, we want to remember the field context vector INLINEFORM2 for a few timesteps  One way of ensuring this is to use a remember (or forget) gate as given below which remembers the previous context vector when required and forgets it when it is time to move on from that field  DISPLAYFORM0
where INLINEFORM0 are parameters to be learned  The job of the forget gate is to ensure that INLINEFORM1 is similar to INLINEFORM2 when required (i e , by learning INLINEFORM3 when we want to continue focusing on the same field) and different when it is time to move on (by learning that INLINEFORM4 ) 
Next, the never look back property implies that once we have moved away from a field we are unlikely to pay attention to it again  For example, once we have rendered all the occupations in the generated description there is no need to return back to the occupation field  In other words, once we have moved on ( INLINEFORM0 ), we want the successive field context vectors INLINEFORM1 to be very different from the previous field vectors INLINEFORM2   One way of ensuring this is to orthogonalize successive field vectors using DISPLAYFORM0
where INLINEFORM0 is the dot product between vectors INLINEFORM1 and INLINEFORM2   The above equation essentially subtracts the component of INLINEFORM3 along INLINEFORM4   INLINEFORM5 is a learned parameter which controls the degree of orthogonalization thereby allowing a soft orthogonalization (i e , the entire component along INLINEFORM6 is not subtracted but only a fraction of it)  The above equation only ensures that INLINEFORM7 is soft-orthogonal to INLINEFORM8   Alternately, we could pass the sequence of context vectors, INLINEFORM9 generated so far through a GRU cell  The state of this GRU cell at each time step would thus be aware of the history of the field vectors till that timestep  Now instead of orthogonalizing INLINEFORM10 to INLINEFORM11 we could orthogonalize INLINEFORM12 to the hidden state of this GRU at time-step INLINEFORM13   In practice, we found this to work better as it accounts for all the field vectors in the history instead of only the previous field vector 
In summary, Equation provides a mechanism for remembering the current field vector when appropriate (thus capturing stay-on behavior) using a remember gate  On the other hand, Equation EQREF10 explicitly ensures that the field vector is very different (soft-orthogonal) from the previous field vectors once it is time to move on (thus capturing never look back behavior)  The value of INLINEFORM0 computed in Equation EQREF10 is then used in Equation   The INLINEFORM1 (macro) thus obtained is then concatenated with INLINEFORM2 (micro) and fed to the decoder (see Fig  FIGREF3 )
Experimental setup
We now describe our experimental setup:
Datasets
We use the WikiBio dataset introduced by lebret2016neural  It consists of INLINEFORM0 biography articles from English Wikipedia  A biography article corresponds to a person (sportsman, politician, historical figure, actor, etc )  Each Wikipedia article has an accompanying infobox which serves as the structured input and the task is to generate the first sentence of the article (which typically is a one-line description of the person)  We used the same train, valid and test sets which were made publicly available by lebret2016neural 
We also introduce two new biography datasets, one in French and one in German  These datasets were created and pre-processed using the same procedure as outlined in lebret2016neural  Specifically, we extracted the infoboxes and the first sentence from the corresponding Wikipedia article  As with the English dataset, we split the French and German datasets randomly into train (80%), test (10%) and valid (10%)  The French and German datasets extracted by us has been made publicly available  The number of examples was 170K and 50K and the vocabulary size was 297K and 143K for French and German respectively  Although in this work we focus only on generating descriptions in one language, we hope that this dataset will also be useful for developing models which jointly learn to generate descriptions from structured data in multiple languages 
Models compared
We compare with the following models:
1  BIBREF0 : This is a conditional language model which uses a feed-forward neural network to predict the next word in the description conditioned on local characteristics (i e , words within a field) and global characteristics (i e , overall structure of the infobox) 
2  BIBREF9 : This model was proposed in the context of the WeatherGov and RoboCup datasets which have a much smaller vocabulary  They use an improved attention model with additional regularizer terms which influence the weights assigned to the fields 
3  Basic Seq2Seq: This is the vanilla encode-attend-decode model BIBREF1   Further, to deal with the large vocabulary ( INLINEFORM0 400K words) we use a copying mechanism as a post-processing step  Specifically, we identify the time steps at which the decoder produces unknown words (denoted by the special symbol UNK)  For each such time step, we look at the attention weights on the input words and replace the UNK word by that input word which has received maximum attention at this timestep  This process is similar to the one described in BIBREF30   Even lebret2016neural have a copying mechanism tightly integrated with their model 
Hyperparameter tuning
We tuned the hyperparameters of all the models using a validation set  As mentioned earlier, we used a bidirectional GRU cell as the function INLINEFORM0 for computing the representation of the fields and the values (see Section SECREF4 )  For all the models, we experimented with GRU state sizes of 128, 256 and 512  The total number of unique words in the corpus is around 400K (this includes the words in the infobox and the descriptions)  Of these, we retained only the top 20K words in our vocabulary (same as BIBREF0 )  We initialized the embeddings of these words with 300 dimensional Glove embeddings BIBREF31   We used Adam BIBREF32 with a learning rate of INLINEFORM1 , INLINEFORM2 and INLINEFORM3   We trained the model for a maximum of 20 epochs and used early stopping with the patience set to 5 epochs 
Results and Discussions
We now discuss the results of our experiments 
Comparison of different models
Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics  We first make a few observations based on the results on the English dataset (Table TABREF15 )  The basic seq2seq model, as well as the model proposed by weather16, perform better than the model proposed by lebret2016neural  Our final model with bifocal attention and gated orthogonalization gives the best performance and does 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method BIBREF0   In Table TABREF16 , we show some qualitative examples of the output generated by different models 
Human Evaluations
To make a qualitative assessment of the generated sentences, we conducted a human study on a sample of 500 Infoboxes which were sampled from English dataset  The annotators for this task were undergraduate and graduate students  For each of these infoboxes, we generated summaries using the basic seq2seq model and our final model with bifocal attention and gated orthogonalization  For each description and for each model, we asked three annotators to rank the output of the systems based on i) adequacy (i e  does it capture relevant information from the infobox), (ii) fluency (i e  grammar) and (iii) relative preference (i e , which of the two outputs would be preferred)  Overall the average fluency/adequacy (on a scale of 5) for basic seq2seq model was INLINEFORM0 and INLINEFORM1 for our model respectively 
The results from Table TABREF17 suggest that in general gated orthogonalization model performs better than the basic seq2seq model  Additionally, annotators were asked to verify if the generated summaries look natural (i e, as if they were generated by humans)  In 423 out of 500 cases, the annotators said “Yes” suggesting that gated orthogonalization model indeed produces good descriptions 
Performance on different languages
The results on the French and German datasets are summarized in Tables TABREF20 and TABREF20 respectively  Note that the code of BIBREF0 is not publicly available, hence we could not report numbers for French and German using their model  We observe that our final model gives the best performance - though the bifocal attention model performs poorly as compared to the basic seq2seq model on French  However, the overall performance for French and German are much smaller than those for English  There could be multiple reasons for this  First, the amount of training data in these two languages is smaller than that in English  Specifically, the amount of training data available in French (German) is only INLINEFORM0 ( INLINEFORM1 )% of that available for English  Second, on average the descriptions in French and German are longer than that in English (EN: INLINEFORM2 words, FR: INLINEFORM3 words and DE: INLINEFORM4 words)  Finally, a manual inspection across the three languages suggests that the English descriptions have a more consistent structure than the French descriptions  For example, most English descriptions start with name followed by date of birth but this is not the case in French  However, this is only a qualitative observation and it is hard to quantify this characteristic of the French and German datasets 
Visualizing Attention Weights
If the proposed model indeed works well then we should see attention weights that are consistent with the stay on and never look back behavior  To verify this, we plotted the attention weights in cases where the model with gated orthogonalization does better than the model with only bifocal attention  Figure FIGREF21 shows the attention weights corresponding to infobox in Figure FIGREF25   Notice that the model without gated orthogonalization has attention on both name field and article title while rendering the name  The model with gated orthogonalization, on the other hand, stays on the name field for as long as it is required but then moves and never returns to it (as expected) 
Due to lack of space, we do not show similar plots for French and German but we would like to mention that, in general, the differences between the attention weights learned by the model with and without gated orthogonalization were more pronounced for the French/German dataset than the English dataset  This is in agreement with the results reported in Table TABREF20 and TABREF20 where the improvements given by gated orthogonalization are more for French/German than for English 
Out of domain results
What if the model sees a different INLINEFORM0 of person at test time? For example, what if the training data does not contain any sportspersons but at test time we encounter the infobox of a sportsperson  This is the same as seeing out-of-domain data at test time  Such a situation is quite expected in the products domain where new products with new features (fields) get frequently added to the catalog  We were interested in three questions here  First, we wanted to see if testing the model on out-of-domain data indeed leads to a drop in the performance  For this, we compared the performance of our best model in two scenarios (i) trained on data from all domains (including the target domain) and tested on the target domain (sports, arts) and (ii) trained on data from all domains except the target domain and tested on the target domain  Comparing rows 1 and 2 of Table TABREF32 we observed a significant drop in the performance  Note that the numbers for sports domain in row 1 are much better than the Arts domain because roughly 40% of the WikiBio training data contains sportspersons 
Next, we wanted to see if we can use a small amount of data from the target domain to fine tune a model trained on the out of domain data  We observe that even with very small amounts of target domain data the performance starts improving significantly (see rows 3 and 4 of Table TABREF32 )  Note that if we train a model from scratch with only limited data from the target domain instead of fine-tuning a model trained on a different source domain then the performance is very poor  In particular, training a model from scratch with 10K training instances we get a BLEU score of INLINEFORM0 and INLINEFORM1 for arts and sports respectively  Finally, even though the actual words used for describing a sportsperson (footballer, cricketer, etc ) would be very different from the words used to describe an artist (actor, musician, etc ) they might share many fields (for example, date of birth, occupation, etc )  As seen in Figure FIGREF28 (attention weights corresponding to the infobox in Figure FIGREF27 ), the model predicts the attention weights correctly for common fields (such as occupation) but it is unable to use the right vocabulary to describe the occupation (since it has not seen such words frequently in the training data)  However, once we fine tune the model with limited data from the target domain we see that it picks up the new vocabulary and produces a correct description of the occupation 
Conclusion
We present a model for generating natural language descriptions from structured data  To address specific characteristics of the problem we propose neural components for fused bifocal attention and gated orthogonalization to address stay on and never look back behavior while decoding  Our final model outperforms an existing state of the art model on a large scale WikiBio dataset by 21%  We also introduce datasets for French and German and demonstrate that our model gives state of the art results on these datasets  Finally, we perform experiments with an out-of-domain model and show that if such a model is fine-tuned with small amounts of in domain data then it can give an improved performance on the target domain 
Given the multilingual nature of the new datasets, as future work, we would like to build models which can jointly learn to generate natural language descriptions from structured data in multiple languages  One idea is to replace the concepts in the input infobox by Wikidata concept ids which are language agnostic  A large amount of input vocabulary could thus be shared across languages thereby facilitating joint learning 
Acknowledgements
We thank Google for supporting Preksha Nema through their Google India Ph D  Fellowship program  We also thank Microsoft Research India for supporting Shreyas Shetty through their generous travel grant for attending the conference 

ground_truth: Yes

Yes