dataesr/scientific-paragraphs-categorization huggingface dataset:
Out-of-domain evaluation datasets To evaluate if models learn about emotional expression beyond the domain of Vent as a social platform, we include five OOD datasets with emotion labels and texts associated with the emotions. The OOD datasets are the following: • enISEAR  [17]  is a dataset of emotional event descriptions in English using the International Survey on Emotion Antecedents and Reactions (ISEAR) approach  [39]  via crowdsourcing. Annotators generated event-focused emotion descriptions using the template: "I felt [emotion] when/because [situation]". While the study included annotations by readers, we only use the annotation of the author of the text to evaluate models. The dataset consists of 1001 instances for seven emotions, four of which match our emotion labels to provide an out-of-domain test.
We design the task as a prediction of the text in which we have replaced the emotion word with the placeholder mask, which is a special token common in language models to denote a missing word. enISEAR is generated by asking participants to describe an emotion-inducing situation, a design that limits its external validity with respect to social media but that has the highest standard of internal validity with text annotations produced in a controlled setup. We consider enISEAR as the out-of-domain dataset most relevant to test the psychological validity of the emotion detection of models, while other datasets from social media are necessary to evaluate models in other domains once this psychological validity level is clear. • GoEmotions  [24]  is a corpus of English comments extracted from Reddit with manual annotations for multiple emotions.
It is a reader-labeled emotion dataset with labels assigned when at least three annotators gave the same label to a comment. For our out-of-domain test, we include the subset of the test split with a single label from among the Ekman category of the dataset, thus having Sadness, Anger, Fear, and Joy as a general positive emotion label. • TEC  [40]  is a corpus of tweets posted between Nov. 15, 2011 and Dec. 6, 2011 with self-label for emotions using emotion-word hashtags. The hashtags serve as the emotion label for classification and are removed from the tweet texts. We sample 10% of the dataset at random as our out-of-domain test set. Since the hashtags are assigned by the authors of the tweets, the dataset can be considered labeled from the perspective of the writer. • Universal Joy  [34]  is a collection of anonymized public Facebook posts in 18 languages labeled with five emotions: anger, anticipation, fear, joy, and sadness.
The labels are derived from the Facebook "feelings tag" provided by the writers of the posts. We use the English subset of the test set for our analysis. • SemEval  [41]  is a collection of tweets in three languages from 2016 and 2017 collected from Twitter using emotion keywords as queries. Subsequently, matching tweets were annotated by crowdworkers for emotion intensity, valence, and basic emotion classes. This dataset was the benchmark data for the competition about affect detection in SemEval. Here, we use the test data by including only instances with a single label that correspond to one of the labels in our model. Note that for the OOD datasets (GoEmotions, TEC, Universal Joy, and SemEval), we use only the test sample for OOD evaluation and exclude other training or development samples.
We do this to provide an evaluation that can be compared to previous and future supervised methods that use the training samples. Based on our selection criteria, we find only 11 tweets with the Affection label in the SemEval dataset. So, we consider Happiness and Affection to be the Happiness emotion label, which limits the nuance in which we can assess classifications within positive emotions in out-of-domain settings but still enables a wider differentiation between general positive emotions and three negative emotions. Descriptive statistics of the counts and proportions of labels in the five datasets can be found in Table  3. We use the in-domain and OOD datasets to evaluate the performance of models in our experimental setup. We calculate the macro-averaged F1 score over all emotion labels and report results with the F1 score of each of the emotion labels, as their frequencies greatly differ in several of the datasets we use for evaluation.
Models Model design and pre-training Pre-trained language models have shown state-of-theart performance on many natural language processing tasks. We expect language models pre-trained on social media data to perform better on the Vent dataset. In preliminary experiments using performance on the development set, we test three pre-trained models based on the Robustly optimized BERT approach (RoBERTa) architecture and pretraining: Roberta-base  [42] , Twitter-RoBERTa  [23] , and BERTweet-base  [25]. BERTweetbase had the best performance on the development set and thus we chose to continue our work with BERTweet-base and its large version, BERTweet-large, in all our experiments. BERTweet-base and BERTweet-large are transformers model pre-trained on 850M tweets with 12 and 24 layers, respectively.
Model fine-tuning with labeled data We implement a multiclass classifier for the five emotion labels: Anger, Fear, Sadness, Happiness, and Affection. We train classifiers starting from LEIA-LM-base and LEIA-LM-large using a two-step approach. First, we perform linear probing to initialize the classifier head and then full fine-tuning of the model. For linear probing, only the classifier head is randomly initialized and trained on the training dataset while the remaining model parameters are fixed. This initial step can be seen as a way to align the features of the prediction head and the base model to minimize feature distortion  [32]. In the subsequent full fine-tuning step, the prediction head is initialized from the parameters learned from the initial linear probing step. We also fine-tune a BERTweet-base and a BERTweet-large model without the eMLM step.
Polarity benchmark The discrete emotions that we consider can be grouped by valence into positive polarity (happiness and affection) and negative polarity (anger, fear, and sadness). We examine the effectiveness of post-processing the prediction from our models into a polarity classifier by taking the sum of the probability for happiness and affection as the positive polarity score. We then compare this strategy to two well-known approaches for polarity classification of online content: LIWC and VADER  [51]. We use the Tone score from LIWC and the compound score from VADER as the polarity score, respectively. For each method, we compute the Area Under the Receiver Operating Characteristics Curve (AUC) for the prediction of positive/negative polarity. Comparison with GPT models Large generative models provide the opportunity for the classification of emotions in social media text in emotion classes as our models, but rate limits and pricing hinder an exhaustive evaluation with our full Vent test datasets, which contain nearly 3 million posts.
To provide a comparison against OpenAI's GPT-3. 5 and GPT-4 models, we design an additional in-domain analysis based on a random sample of 1000 posts from each emotion category in the user test split of the Vent dataset. For an outof-domain comparison, we ran both GPT models over the full test samples of the OOD datasets. We prompt GPT models with instructions to output only one of the emotion categories for each input text via the OpenAI chat/completions endpoint with a temperature of 0. For cases, where the model did not return the expected output we repeatedly prompt the model to output one of the emotion categories. In addition, we limit the number of output tokens to 3 (the maximum number of tokens required by the Generative Pretrained Transformer (GPT) tokenizer to cover all emotion categories).
This shows that the general performance of LEIA-base and LEIA-large is not as a result of   2  Results within the Vent dataset in the three test samples. Error bars show bootstrap 95% confidence intervals and may be too small to be visible due to the large sample sizes bias from higher performance on majority class. The only class that has a slightly lower F1 is Fear, but LEIA-base and LEIA-large still outperform all other methods on it. One observation is that NBSVM also performs slightly worse for Fear than for other emotions in contrast with LIWC, which obtains a comparatively better performance than NRC in the Fear category.
Out-of-domain results Our out-of-domain benchmark shows that LEIA can detect emotional states in other types of text and social media platforms beyond Vent. Table  5  shows the Macro-F1 scores for the five out-of-domain test sets. LEIA-base and LEIA-large have significantly higher F1 scores than all other methods when evaluated on 4 out of the 5 OOD datasets. The NBSVM has a comparable performance in the GoEmotions dataset, where the F1 of NBSVM and of LEIA-base are not significantly different. We also observe that a larger model does not necessarily lead to better performance on OOD datasets, as LEIA-large only shows a substantially different performance on the enISEAR dataset. Figure  3  shows the F1 score for each class on the OOD datasets. In general, LEIA often outperforms baselines across labels.
LEIA is significantly better than the baselines for Happiness and Sadness in the Universal Joy and TEC datasets, for all emotions in the enISEAR dataset, and for all emotions except Fear and Sadness in the SemEval dataset. On the GoEmotions dataset, LEIA is tied with NBSVM as the best method to detect Anger as F1 score is not significantly different. The Fear class evaluation poses some challenges in this OOD evaluation since evaluation samples for this class can be very small (e. g. 11 posts in Universal Joy and 77 in GoEmotions). In the case of Fear, LIWC performs significantly better than the supervised approaches on GoEmotions, SemEval, and TEC. Recall that the dictionary approach is based on a binary classification setting which is easier than a multiclass classification setting. Despite this, the performance of the dictionary approach is significantly lower for Happiness.
This trend is similar to the performance observed on the in-domain test sets. We can conclude that LEIA shows a good generalization beyond the domain it was trained on, first by achieving very high performance in enISEAR, the test closest to psychological methodology, but also achieving good performance for datasets that include posts from other social media such as Twitter and Facebook. The lower performance recorded for Fear on the out-of-domain test sets is not surprising as the model performance on this category tends to be lower on the in-domain test sets too. LEIA achieves a consistently high score for Happiness on the out-of-domain test sets despite the fact that it is one of the least frequent categories in the training set. This suggests that it constitutes an easier category for the model to recognize across domains than more nuanced negative emotions.
Comparison with GPT-3. 5 and GPT-4 models Table  6  shows the performance comparison of LEIA-base, LEIA-large, GPT-3. 5, and GPT-4 on a sample of 1000 examples for each emotion category drawn from the user test split of the Vent dataset. LEIA-base and LEIA-large perform better on all emotion classes of the Vent dataset than GPT-3. 5 and GPT-4. We show the performance comparison on the out-of-domain datasets in Table  7. The F1 score per emotion category on each dataset is in Fig. 5  in the Appendix. GPT-3. 5 and GPT-4 perform better than both LEIA-base and LEIA-large on the OOD datasets. Our results are consistent with recent findings showing that smaller models tailored for specific tasks perform better than large generative models such as GPT-3. 5 and GPT-4, especially when evaluated against datasets that are unlikely to be part of the training data of GPT models  [52, 53].
A visible trend from Table  7  is that GPT-3. 5 and GPT-4 models show higher performance on datasets on which LEIA-base and LEIA-large show relatively higher performance and vice-versa. This may point to the level of difficulty of some of the OOD datasets. Although GPT-3. 5 and GPT-4 models perform better than our models on the OOD datasets, we do not know whether this performance is clearly a capability of the model or due to data contamination as these models are trained on massive datasets which may include benchmark datasets  [54]. Moreover, it has also been documented that it is challenging for large language models to infer mental state from textual data  [55]. As noted by the authors of  [53] , an avenue for future work is to explore approaches that   combine large generative models with smaller domain-specific models that can be applied efficiently and at scale.
Polarity classification benchmark Results on the out-of-domain datasets can be found in Table  8. On 4 out of the 5 out-of-domain datasets LEIA-base and/or LEIA-large perform better than the dictionary baselines: LIWC and VADER. LIWC and VADER show better performance only on the GoEmotions dataset. Comparing LIWC and VADER, we find that VADER performs better than LIWC on enISEAR, GoEmotions, and SemEval while LIWC is superior on Universal Joy and TEC. Error analysis We examine a random sample of 50 incorrect predictions from the user test split (10 per label) of the Vent dataset. We find that majority of errors in the sample can be categorized into the following cases: 1. Messages conveying an expectation of a positive outcome while the self-assigned label has negative valence (e. g. , I need a good online game).
We compute the recall for each tag from the prediction of one of our models, LEIA-base, on the user test split of the Vent dataset. We consider the prediction a hit for a given tag when the model makes the correct emotion prediction and a miss otherwise. We find that the recall for each tag within each group is comparable ranging from about 0. 6 to 0. 8, which suggests that our grouping is reasonable. The details of the recall score for each tag are in Fig. 6  in the Appendix.
Feature attributions We examine the salient features that contribute to the predictions made by LEIA-base on a set of examples from the enISEAR dataset. We apply the Local Interpretable Model-agnostic Explanations (LIME) method for model interpretability  [56] , We observe that for the first example, the model incorrectly predicts Affection as the most likely label where the true label is Happiness, which is an error of a weaker kind since enISEAR does not have an Affection label and both emotions are close in terms of valence. The second highest class is Happiness and the prediction is positively based on words expressing high arousal and valence (e. g. , "incredible") and negatively based on the word "worrying". In the second example, the model also seems to use relevant words linked to each other (e.
Discussion We present LEIA, a language model in two sizes (LEIA-base and LEIA-large), that leverages approaches for adapting pre-trained language models for emotion identification. We show that using an emotion lexicon with task-adaptive pre-training, in this case focusing on emotion words, is effective for improving model performance using BERTweet-base and BERTweet-large language models. LEIA generalizes beyond Vent posts as it shows better performance on texts written by users not included in its training data and future time periods. It achieves a balanced performance across emotion labels despite their imbalance in training data and this performance is also seen on out-of-domain texts for the considered emotions except for Fear. These results are in part possible thanks to focusing on a small set of emotions suggested by psychological research, as classifying the larger set of mood labels in Vent  [21]  is a substantially harder task we did not tackle here.
Also, the Vent dataset, which despite being generated on a platform not as large as common ones in research, e. g. Twitter and Reddit, has a sufficiently large scale that enables the models to learn a broader range of emotional expressions. The performance of LEIA-base is comparable to LEIA-large across tests in our benchmark with one notable exception: LEIA-large is substantially better for the enISEAR dataset. This dataset is especially important given the psychological methodology used to generate it, which allows us to compare the results of machine learning methods with self-reported labels in a controlled setup. LEIA's performance in enISEAR is especially high, reaching F1 of 70 for LEIA-base and 79 for LEIA-large, showing a high level of psychological validity, especially when compared to other methods in the benchmark that achieve at most 55.
On the OOD datasets, GPT-3. 5 and GPT-4 perform better than LEIA-base and LEIA-large. This finding is in line with existing findings that show that smaller domain-specific models perform better than larger general-purpose generative models. One issue with assessing the real capability of large language models is the possibility of data contamination where benchmark datasets can potentially be part of their training data. Large language models are often accessible via Application Programming Interfaces (APIs) which make it easy in practice to use with its attendant cost. However, rate limits and financial costs make it a less attractive option to apply at scale. This is even complicated by the need to make repeated calls to the API when the model does not follow the instructions provided in the prompt.
Another limitation is our focus mainly on English posts, providing no evidence here of the potential of this approach for other languages. Furthermore, we study five emotion labels guided by psychological research, but several competing representations models for emotion are available. Humans are able to classify a larger number of basic emotions and can also quantify emotions in dimensional spaces, two open areas that can be explored with more nuanced labeling schemes. While self-annotated datasets have the potential to become the new gold standard beyond crowdworkers, the labeling scheme of the Vent dataset is designed as part of its interface rather than as a psychometric measure applied privately and not visible on the platform. This is still closer to general emotion expression than automatic labeling with emoji or hashtags, but models like LEIA-base or LEIAlarge can be substantially improved with psychological methods like experience sampling  [18]  and with validated psychological scales to measure emotions in dimensional spaces  [57, 58].
This would have the added value of being applicable to studying more nuanced emotion dynamics that need dimensional measurements and not just classification, for example using social media data  [6, 59, 60]. Broader impact and ethical considerations This work shares the same ethical concerns with other emotion recognition systems as highlighted in  [61]. Emotion detection models should be used responsibly and special care should be taken when they are applied in new scenarios, not only because of their possible lower performance but also due to possible different privacy expectations with respect to emotions. We must note that we have no way of estimating the demographic diversity of Vent users and it is very likely that the model misses idiosyncrasies of emotional expression in minority groups and in cultures not represented in the dataset.
Discovering new hyperlinks enables Web crawlers to find new pages that have not yet been indexed. This is especially important for focused crawlers because they strive to provide a comprehensive analysis of specific parts of the Web, thus prioritizing discovery of new pages over discovery of changes in content. In the literature, changes in hyperlinks and content have been usually considered simultaneously. However, there is also evidence suggesting that these two types of changes are not necessarily related. Moreover, many studies about predicting changes assume that long history of a page is available, which is unattainable in practice. The aim of this work is to provide a methodology for detecting new hyperlinks effectively using a short history. To this end, we use a dataset of ten crawls at intervals of one week.
We classify the selected features in four categories: the static page (SP) features, such as the content of a page  [21, 8, 44, 2, 37, 40, 41] ; the dynamic page (DP) features that represent the (short) history of SP  [44, 12, 8, 15, 33, 43, 29, 4, 37, 21] , and the number of new hyperlinks in the past crawls; the static network (SN) features that include incoming hyperlinks, TrustRank, and the SP features of content-related pages  [39, 44, 21, 16] ; the dynamic network (DN) features, which are the (short) history of SN (excluding related page features) and the weighted average link change rate of related pages. Our conclusion, based on the results obtained from our dataset, is that both DP and DN features are highly informative. In fact, the number of new outlinks in the previous time period (a DP feature) turns out to be a strong baseline for predicting new outlinks.
This paper is organized as follows. The problem description is presented in Section 2. Section 3 explains how the dataset is obtained. Section 4 covers statistical properties of the number of new outlinks. In Section 5 we present our prediction methods including three prediction targets, the sets of features, and learning methods. Section 6 presents the results on predictive power of static features and dynamic features. In Section 7 we propose ranking metrics based on predicted emergence of new outlinks, and evaluate their performance against baselines and ground truth targets. In Section 8 we summarize the main conclusions and suggest future research directions. Lastly, an overview of related works is given in Section 9.
Dataset For this research, we perform ten crawls at intervals of one week, starting 13-07-2020 till 14-09-2020. These choices were based on the practical experience and application in focused crawling, where weekly recrawls proved to strike a good balance between the freshness of the index and available resources, while long history requires infeasible storage capacity, and is simply unavailable for newly found pages. Formally, in our data we have n = 10, and we consider 9 one-week intervals [t i , t i+1 ), i = 1, 2,. , 9. The predictions are made for week 9, and the DP and DN features are computed based on the history of at most 8 weeks. For each page p at time t we obtain data that include the URLs on the page, the fetching date, URLs that link to p, the PageRank, TrustRank, and the semantic vector sem(p(t)) that describes the content of the page.
In Section 5 we will explain in more detail how PageRank, TrustRank and the semantic vector are obtained. For collecting our dataset, the seeds are 93,684 unique URLs selected as follows: from more than a billion URLs of a previous large crawl, 100,000 URLs have been randomly selected, then truncated to the home page. The crawl is guided by PageRank×TrustRank in which TrustRank is biased toward renowned websites that are historically more Euro-and US-centric. At each crawl, we stopped the crawling process when one million pages have been collected. Many pages have been discarded, for example, pages that did not contain the entire timeline of 10 crawls, or missed some data, for instance, the semantic vector. This resulted in the dataset that contains 384,323 pages with complete information included.