UNIST-Eunchan/NLP-Paper-to-QA-Generation huggingface dataset:
Abstract: This paper describes the systems submitted by the department of electronic engineering, institute of microelectronics of Tsinghua university and TsingMicro Co. Ltd. (THUEE) to the NIST 2019 speaker recognition evaluation CTS challenge. Six subsystems, including etdnn/ams, ftdnn/as, eftdnn/ams, resnet, multitask and c-vector are developed in this evaluation.

Introduction: This paper describes the systems developed by the department of electronic engineering, institute of microelectronics of Tsinghua university and TsingMicro Co. Ltd. (THUEE) for the NIST 2019 speaker recognition evaluation (SRE) CTS challenge BIBREF0. Six subsystems, including etdnn/ams, ftdnn/as, eftdnn/ams, resnet, multitask and c-vector are developed in this evaluation. All the subsystems consists of a deep neural network followed by dimension deduction, score normalization and calibration. For each system, we begin with a summary of the data usage, followed by a description of the system setup along with their hyperparameters. Finally, we report experimental results obtained by each subsystem and fusion system on the SRE18 development and SRE18 evaluation datasets.

Question: What dataset was used in this challenge?
Answer: SRE04/05/06/08/10/MIXER6
LDC98S75/LDC99S79/LDC2002S06/LDC2001S13/LDC2004S07
Voxceleb 1/2
Fisher + Switchboard I
Callhome+Callfriend
Abstract: In this paper we present the approach of introducing thesaurus knowledge into probabilistic topic models. The main idea of the approach is based on the assumption that the frequencies of semantically related words and phrases, which are met in the same texts, should be enhanced: this action leads to their larger contribution into topics found in these texts. We have conducted experiments with several thesauri and found that for improving topic models, it is useful to utilize domain-specific knowledge. If a general thesaurus, such as WordNet, is used, the thesaurus-based improvement of topic models can be achieved with excluding hyponymy relations in combined topic models.

Introduction: Currently, probabilistic topic models are important tools for improving automatic text processing including information retrieval, text categorization, summarization, etc. Besides, they can be useful in supporting expert analysis of document collections, news flows, or large volumes of messages in social networks BIBREF0 , BIBREF1 , BIBREF2 . To facilitate this analysis, such approaches as automatic topic labeling and various visualization techniques have been proposed BIBREF1 , BIBREF3 . Boyd-Graber et al. BIBREF4 indicate that to be understandable by humans, topics should be specific, coherent, and informative. Relationships between the topic components can be inferred. In BIBREF1 four topic visualization approaches are compared. The authors of the experiment concluded that manual topic labels include a considerable number of phrases; users prefer shorter labels with more general words and tend to incorporate phrases and more generic terminology when using more complex network graph. Blei and Lafferty BIBREF3 visualize topics with ngrams consisting of words mentioned in these topics. These works show that phrases and knowledge about hyponyms/hypernyms are important for topic representation. In this paper we describe an approach to integrate large manual lexical resources such as WordNet or EuroVoc into probabilistic topic models, as well as automatically extracted n-grams to improve coherence and informativeness of generated topics. The structure of the paper is as follows. In Section 2 we consider related works. Section 3 describes the proposed approach. Section 4 enumerates automatic quality measures used in experiments. Section 5 presents the results obtained on several text collections according to automatic measures. Section 6 describes the results of manual evaluation of combined topic models for Islam Internet-site thematic analysis.

Question: Do they reduce language variation of text by enhancing frequencies?
Answer: Variation decreases when frequencies of synonyms is enhanced; variation increases when frequencies of synonyms, hyponyms, hypernyms are enhanced
Abstract: Pretrained transformer-based language models have achieved state of the art across countless tasks in natural language processing. These models are highly expressive, comprising at least a hundred million parameters and a dozen layers. Recent evidence suggests that only a few of the final layers need to be fine-tuned for high quality on downstream tasks. Naturally, a subsequent research question is, "how many of the last layers do we need to fine-tune?" In this paper, we precisely answer this question. We examine two recent pretrained language models, BERT and RoBERTa, across standard tasks in textual entailment, semantic similarity, sentiment analysis, and linguistic acceptability. We vary the number of final layers that are fine-tuned, then study the resulting change in task-specific effectiveness. We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality. Surprisingly, we also find that fine-tuning all layers does not always help.

Introduction: Transformer-based pretrained language models are a battle-tested solution to a plethora of natural language processing tasks. In this paradigm, a transformer-based language model is first trained on copious amounts of text, then fine-tuned on task-specific data. BERT BIBREF0, XLNet BIBREF1, and RoBERTa BIBREF2 are some of the most well-known ones, representing the current state of the art in natural language inference, question answering, and sentiment classification, to list a few. These models are extremely expressive, consisting of at least a hundred million parameters, a hundred attention heads, and a dozen layers. An emerging line of work questions the need for such a parameter-loaded model, especially on a single downstream task. BIBREF3, for example, note that only a few attention heads need to be retained in each layer for acceptable effectiveness. BIBREF4 find that, on many tasks, just the last few layers change the most after the fine-tuning process. We take these observations as evidence that only the last few layers necessarily need to be fine-tuned. The central objective of our paper is, then, to determine how many of the last layers actually need fine-tuning. Why is this an important subject of study? Pragmatically, a reasonable cutoff point saves computational memory across fine-tuning multiple tasks, which bolsters the effectiveness of existing parameter-saving methods BIBREF5. Pedagogically, understanding the relationship between the number of fine-tuned layers and the resulting model quality may guide future works in modeling. Our research contribution is a comprehensive evaluation, across multiple pretrained transformers and datasets, of the number of final layers needed for fine-tuning. We show that, on most tasks, we need to fine-tune only one fourth of the final layers to achieve within 10% parity with the full model. Surprisingly, on SST-2, a sentiment classification dataset, we find that not fine-tuning all of the layers leads to improved quality.

Question: Do they test against the large version of RoBERTa?
Answer: For GLUE bencmark no, for dataset MRPC, SST-B, SST-2 and COLA yes.
Abstract: Natural language inference (NLI) aims at predicting the relationship between a given pair of premise and hypothesis. However, several works have found that there widely exists a bias pattern called annotation artifacts in NLI datasets, making it possible to identify the label only by looking at the hypothesis. This irregularity makes the evaluation results over-estimated and affects models' generalization ability. In this paper, we consider a more trust-worthy setting, i.e., cross-dataset evaluation. We explore the impacts of annotation artifacts in cross-dataset testing. Furthermore, we propose a training framework to mitigate the impacts of the bias pattern. Experimental results demonstrate that our methods can alleviate the negative effect of the artifacts and improve the generalization ability of models.

Introduction: Natural language inference (NLI) is a widely-studied problem in natural language processing. It aims at comparing a pair of sentences (i.e. a premise and a hypothesis), and inferring the relationship between them (i.e., entailment, neutral and contradiction). Large-scaled datasets like SNLI BIBREF0 and MultiNLI BIBREF1 have been created by crowd-sourcing and fertilized NLI research substantially. However, several works BIBREF2, BIBREF3, BIBREF4 have pointed out that crowd-sourcing workers have brought a bias pattern named annotation artifacts in these NLI datasets. Such artifacts in hypotheses can reveal the labels and make it possible to predict the labels solely by looking at the hypotheses. For example, models trained on SNLI with only the hypotheses can achieve an accuracy of 67.0%, despite the always predicting the majority-class baseline is only 34.3% BIBREF2. Classifiers trained on NLI datasets are supposed to make predictions by understanding the semantic relationships between given sentence pairs. However, it is shown that models are unintentionally utilizing the annotation artifacts BIBREF4, BIBREF2. If the evaluation is conducted under a similar distribution as the training data, e.g., with the given testing set, models will enjoy additional advantages, making the evaluation results over-estimated. On the other hand, if the bias pattern cannot be generalized to the real-world, it may introduce noise to models, thus hurting the generalization ability. In this paper, we use cross-dataset testing to better assess models' generalization ability. We investigate the impacts of annotation artifacts in cross-dataset testing. Furthermore, we propose an easy-adopting debiasing training framework, which doesn't require any additional data or annotations, and apply it to the high-performing Densely Interactive Inference Network BIBREF5. Experiments show that our method can effectively mitigate the bias pattern and improve the cross-dataset generalization ability of models. To the best of our knowledge, our work is the first attempt to alleviate the annotation artifacts without any extra resources.

Question: How does the proposed training framework mitigate the bias pattern?
Answer: by balancing or, smoothing the artifacts across different classes by assigning specific weights for every sample
Abstract: Previous work in Indonesian part-of-speech (POS) tagging are hard to compare as they are not evaluated on a common dataset. Furthermore, in spite of the success of neural network models for English POS tagging, they are rarely explored for Indonesian. In this paper, we explored various techniques for Indonesian POS tagging, including rule-based, CRF, and neural network-based models. We evaluated our models on the IDN Tagged Corpus. A new state-of-the-art of 97.47 F1 score is achieved with a recurrent neural network. To provide a standard for future work, we release the dataset split that we used publicly.

Introduction: Part-of-speech (POS) tagging is a process to tag tokens in a string with their corresponding part-of-speech (e.g., noun, verb, etc). POS tagging is considered as one of the most basic tasks in NLP, as it is usually the first component in an NLP pipeline. This is because POS tags are shown to be useful features in various NLP tasks, such as named entity recognition BIBREF0 , BIBREF1 , machine translation BIBREF2 , BIBREF3 and constituency parsing BIBREF4 . Therefore, for any language, building a successful NLP system usually requires a well-performing POS tagger. There are quite a number of research on Indonesian POS tagging BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 . However, almost all of them are not evaluated on a common dataset. Even when they are, their train-test split are not the same. This lack of a common benchmark dataset makes a fair comparison among these works difficult. Moreover, despite the success of neural network models for English POS tagging BIBREF9 , BIBREF10 , the use of neural networks is generally unexplored for Indonesian. As a result, published results may not reflect the actual state-of-the-art performance of Indonesian POS tagger. In this work, we explored different neural network architectures for Indonesian POS tagging. We evaluated our experiments on the IDN Tagged Corpus BIBREF11 . Our best model achieves 97.47 INLINEFORM0 score, a new state-of-the-art result for Indonesian POS tagging on the dataset. We release the dataset split that we used to serve as a benchmark for future work.

Question: what neural network models were explored?
Answer: Feedforward, biLSTM
Abstract: Automatic sarcasm detection methods have traditionally been designed for maximum performance on a specific domain. This poses challenges for those wishing to transfer those approaches to other existing or novel domains, which may be typified by very different language characteristics. We develop a general set of features and evaluate it under different training scenarios utilizing in-domain and/or out-of-domain training data. The best-performing scenario, training on both while employing a domain adaptation step, achieves an F1 of 0.780, which is well above baseline F1-measures of 0.515 and 0.345. We also show that the approach outperforms the best results from prior work on the same target domain.

Introduction: Sarcasm, a creative device used to communicate an intended meaning that is actually the opposite of its literal meaning, is notoriously difficult to convey and interpret through text, in part because doing so relies heavily upon shared contextual understandings that can be marked more easily by altered prosody (e.g., emphasis upon certain words) or non-verbal signals (e.g., rolling one's eyes). It is a complex process even for humans, and in fact an inability to detect sarcasm has been linked with a number of neurocognitive disorders, including dementia BIBREF0 . It is similarly a challenging open task in natural language processing, and has direct implications to a number of other critical application areas, such as sentiment analysis. Most research on automatic sarcasm detection to date has focused on the Twitter domain, which boasts an ample source of publicly-available data, some of which is already self-labeled by users for the presence of sarcasm (e.g., with #sarcasm). However, Twitter is highly informal, space-restricted, and subject to frequent topic fluctuations from one post to the next due to the ebb and flow of current events—in short, it is not broadly representative of most text domains. Thus, sarcasm detectors trained using features designed for maximum Twitter performance are not necessarily transferable to other domains. Despite this, it is desirable to develop approaches that can harness the more generalizable information present in the abundance of Twitter data. In this work, we develop a set of domain-independent features for sarcasm detection and show that the features generally perform well across text domains. Further, we validate that domain adaptation can be applied to sarcasm detection to leverage patterns in out-of-domain training data, even when results from training only on that source domain data are extremely bad (far below baseline results), to improve over training on only the target data or over training on the simply combined dataset. Finally, we make a new dataset of sarcastic and non-sarcastic tweets available online as a resource to other researchers.

Question: what training data was used?
Answer: Twitter product reviews containing exactly one of the trailing hashtags “#sarcasm,” “#happiness,” “#sadness,” “#anger,” “#surprise,” “#fear,” and “#disgust”, and Amazon product reviews from the publicly available sarcasm corpus developed by Filatova FILATOVA12.661.
Abstract: As offensive language has become a rising issue for online communities and social media platforms, researchers have been investigating ways of coping with abusive content and developing systems to detect its different types: cyberbullying, hate speech, aggression, etc. With a few notable exceptions, most research on this topic so far has dealt with English. This is mostly due to the availability of language resources for English. To address this shortcoming, this paper presents the first Greek annotated dataset for offensive language identification: the Offensive Greek Tweet Dataset (OGTD). OGTD is a manually annotated dataset containing 4,779 posts from Twitter annotated as offensive and not offensive. Along with a detailed description of the dataset, we evaluate several computational models trained and tested on this data.

Introduction: In the age of social media, offensive content online has become prevalent in recent years. There are many types of offensive content online such as racist and sexist posts and insults and threats targeted at individuals or groups. As such content increasingly occurs online, it has become a growing issue for online communities. This has come to the attention of social media platforms and authorities underlining the urgency to moderate and deal with such content. Several studies in NLP have approached offensive language identification applying machine learning and deep learning systems on annotated data to identify such content. Researchers in the field have worked with different definitions of offensive language with hate speech being the most studied among these types BIBREF0. BIBREF1 investigate the similarity between these sub-tasks. With a few noteworthy exceptions, most research so far has dealt with English, due to the availability of language resources. This gap in the literature recently started to be addressed with studies on Spanish BIBREF2, Hindi BIBREF3, and German BIBREF4, to name a few. In this paper we contribute in this direction presenting the first Greek annotated dataset for offensive language identification: the Offensive Greek Tweet Dataset (OGTD). OGTD uses a working definition of offensive language inspired by the OLID dataset for English BIBREF5 used in the recent OffensEval (SemEval-2019 Task 6) BIBREF6. In its version, 1.0 OGTD contains nearly 4,800 posts collected from Twitter and manually annotated by a team of volunteers, resulting in a high-quality annotated dataset. We trained a number of systems on this dataset and our best results have been obtained from a system using LSTMs and GRU with attention which achieved 0.89 F1 score.

Question: What are the models tested on the dataset?
Answer: linear SVM, RBF SVM, linear classifier with SGDC, multinomial naive bayes, bernoulli naive bayes, pooled GRU, stacked LSTM with attention, LSTM and GRU with attention, 2d convolution with pooling, GRU with Capsule, LSTM with Capsule and attention, and BERT
Abstract: While sentiment analysis has become an established field in the NLP community, research into languages other than English has been hindered by the lack of resources. Although much research in multi-lingual and cross-lingual sentiment analysis has focused on unsupervised or semi-supervised approaches, these still require a large number of resources and do not reach the performance of supervised approaches. With this in mind, we introduce two datasets for supervised aspect-level sentiment analysis in Basque and Catalan, both of which are under-resourced languages. We provide high-quality annotations and benchmarks with the hope that they will be useful to the growing community of researchers working on these languages.

Introduction: Sentiment analysis has become an established field with a number of subfields (aspect-level sentiment analysis, social media sentiment analysis, cross-lingual sentiment analysis), all of which require some kind of annotated resource, either to train a machine-learning based classifier or to test the performance of proposed approaches. Although much research into multi-lingual and cross-lingual sentiment analysis has focused on unsupervised or semi-supervised approaches BIBREF0 , BIBREF1 , BIBREF2 , these techniques still require certain resources (linked wordnets, seed lexicon) and do not generally reach the performance of supervised approaches. In English the state-of-the-art for binary sentiment analysis often reaches nearly 90 percent accuracy BIBREF3 , BIBREF4 , BIBREF5 , but for other languages there is a marked drop in accuracy. This is mainly due to the lack of annotations and resources in these languages. This is especially true of corpora annotated at aspect-level. Unlike document- or tweet-level annotation, aspect-level annotation requires a large amount of effort from the annotators, which further reduces the likelihood of finding an aspect-level sentiment corpus in under-resourced languages. We are, however, aware of one corpus annotated for aspects in German BIBREF6 , although German is not a particularly low-resource language. The movement towards multi-lingual datasets for sentiment analysis is important because many languages offer different challenges, such as complex morphology or highly productive word formation, which can not be overcome by focusing only on English data. The novelty of this work lies in creating corpora which cover both Basque and Catalan languages and are annotated in such a way that they are compatible with similarly compiled corpora available in a number of languages BIBREF7 . This allows for further research into cross-lingual sentiment analysis, as well as introducing the first resource for aspect-level sentiment analysis in Catalan and Basque. The corpus is available at http://hdl.handle.net/10230/33928 or https://jbarnesspain.github.io/resources/.

Question: What is the size of their published dataset?
Answer: 911
Abstract: Poetry generation is an interesting research topic in the field of text generation. As one of the most valuable literary and cultural heritages of China, Chinese classical poetry is very familiar and loved by Chinese people from generation to generation. It has many particular characteristics in its language structure, ranging from form, sound to meaning, thus is regarded as an ideal testing task for text generation. In this paper, we propose a GPT-2 based uniformed framework for generating major types of Chinese classical poems. We define a unified format for formulating all types of training samples by integrating detailed form information, then present a simple form-stressed weighting method in GPT-2 to strengthen the control to the form of the generated poems, with special emphasis on those forms with longer body length. Preliminary experimental results show this enhanced model can generate Chinese classical poems of major types with high quality in both form and content, validating the effectiveness of the proposed strategy. The model has been incorporated into Jiuge, the most influential Chinese classical poetry generation system developed by Tsinghua University (Guo et al., 2019).

Introduction: 1.1em

Question: What is the source of the training/testing data?
Answer: Two major forms(Jueju and Lvshi) of SHI and 121 major forms of CI from Chinese Classical Poerty Corpus (CCPC1.0)