UNIST-Eunchan/NLP-Paper-to-QA-Generation huggingface dataset:
Abstract: Kurdish is a less-resourced language consisting of different dialects written in various scripts. Approximately 30 million people in different countries speak the language. The lack of corpora is one of the main obstacles in Kurdish language processing. In this paper, we present KTC-the Kurdish Textbooks Corpus, which is composed of 31 K-12 textbooks in Sorani dialect. The corpus is normalized and categorized into 12 educational subjects containing 693,800 tokens (110,297 types). Our resource is publicly available for non-commercial use under the CC BY-NC-SA 4.0 license.

Introduction: Kurdish is an Indo-European language mainly spoken in central and eastern Turkey, northern Iraq and Syria, and western Iran. It is a less-resourced language BIBREF0, in other words, a language for which general-purpose grammars and raw internet-based corpora are the main existing resources. The language is spoken in five main dialects, namely, Kurmanji (aka Northern Kurdish), Sorani (aka Central Kurdish), Southern Kurdish, Zazaki and Gorani BIBREF1. Creating lexical databases and text corpora are essential tasks in natural language processing (NLP) development. Text corpora are knowledge repositories which provide semantic descriptions of words. The Kurdish language lacks diverse corpora in both raw and annotated forms BIBREF2, BIBREF3. According to the literature, there is no domain-specific corpus for Kurdish. In this paper, we present KTC, a domain-specific corpus containing K-12 textbooks in Sorani. We consider a domain as a set of related concepts, and a domain-specific corpus as a collection of documents relevant to those concepts BIBREF4. Accordingly, we introduce KTC as a domain-specific corpus because it is based on the textbooks which have been written and compiled by a group of experts, appointed by the Ministry of Education (MoE) of the Kurdistan Region of Iraq, for educational purposes at the K-12 level. The textbooks are selected, written, compiled, and edited by experts in each subject and also by language editors based on a unified grammar and orthography. This corpus was initially collected as an accurate source for developing a Sorani Kurdish spellchecker for scientific writing. KTC contains a range of subjects, and its content is categorized according to those subjects. Given the accuracy of the text from scientific, grammatical, and orthographic points of view, we believe that it is also a fine-grained resource. The corpus will contribute to various NLP tasks in Kurdish, particularly in language modeling and grammatical error correction. In the rest of this paper, Section SECREF2 reviews the related work, Section SECREF3 presents the corpus, Section SECREF4 addresses the challenges in the project and, Section SECREF5 concludes the paper.

Question: what are the off-the-shelf systems discussed in the paper?
Answer: Answer with content missing: (Names of many identifiers missing) TextCat, ChromeCLD, LangDetect, langid.py, whatlang, whatthelang, YALI, LDIG, Polyglot 3000, Lextek Language Identifier and Open Xerox Language Identifier.
Abstract: Recently, there has been interest in multiplicative recurrent neural networks for language modeling. Indeed, simple Recurrent Neural Networks (RNNs) encounter difficulties recovering from past mistakes when generating sequences due to high correlation between hidden states. These challenges can be mitigated by integrating second-order terms in the hidden-state update. One such model, multiplicative Long Short-Term Memory (mLSTM) is particularly interesting in its original formulation because of the sharing of its second-order term, referred to as the intermediate state. We explore these architectural improvements by introducing new models and testing them on character-level language modeling tasks. This allows us to establish the relevance of shared parametrization in recurrent language modeling.

Introduction: One of the principal challenges in computational linguistics is to account for the word order of the document or utterance being processed BIBREF0 . Of course, the numbers of possible phrases grows exponentially with respect to a given phrase length, requiring an approximate approach to summarizing its content. rnn are such an approach, and they are used in various tasks in nlp, such as machine translation BIBREF1 , abstractive summarization BIBREF2 and question answering BIBREF3 . However, rnn, as approximations, suffer from numerical troubles that have been identified, such as that of recovering from past errors when generating phrases. We take interest in a model that mitigates this problem, mrnn, and how it has been and can be combined for new models. To evaluate these models, we use the task of recurrent language modeling, which consists in predicting the next token (character or word) in a document. This paper is organized as follows: rnn and mrnn are introduced respectively in Sections SECREF2 and SECREF3 . Section SECREF4 presents new and existing multiplicative models. Section SECREF5 describes the datasets and experiments performed, as well as results obtained. Sections SECREF6 discusses and concludes our findings.

Question: What models are included in baseline benchmarking results?
Answer: BERT, XLNET RoBERTa, ALBERT, DistilBERT
Abstract: In low-resource settings, the performance of supervised labeling models can be improved with automatically annotated or distantly supervised data, which is cheap to create but often noisy. Previous works have shown that significant improvements can be reached by injecting information about the confusion between clean and noisy labels in this additional training data into the classifier training. However, for noise estimation, these approaches either do not take the input features (in our case word embeddings) into account, or they need to learn the noise modeling from scratch which can be difficult in a low-resource setting. We propose to cluster the training data using the input features and then compute different confusion matrices for each cluster. To the best of our knowledge, our approach is the first to leverage feature-dependent noise modeling with pre-initialized confusion matrices. We evaluate on low-resource named entity recognition settings in several languages, showing that our methods improve upon other confusion-matrix based methods by up to 9%.

Introduction: Most languages, even with millions of speakers, have not been the center for natural language processing and are counted as low-resource for tasks like named entity recognition (NER). Similarly, even for high-resource languages, there exists only few labeled data for most entity types beyond person, location and organization. Distantly- or weakly-supervised approaches have been proposed to solve this issue, e.g., by using lists of entities for labeling raw text BIBREF0, BIBREF1. This allows obtaining large amounts of training data quickly and cheaply. Unfortunately, these labels often contain errors and learning with this noisily-labeled data is difficult and can even reduce overall performance (see, e.g. BIBREF2). A variety of ideas have been proposed to overcome the issues of noisy training data. One popular approach is to estimate the relation between noisy and clean, gold-standard labels and use this noise model to improve the training procedure. However, most of these approaches only assume a dependency between the labels and do not take the features into account when modeling the label noise. This may disregard important information. The global confusion matrix BIBREF3 is a simple model which assumes that the errors in the noisy labels just depend on the clean labels. Our contributions are as follows: We propose to cluster the input words with the help of additional, unlabeled data. Based on this partition of the feature space, we obtain different confusion matrices that describe the relationship between clean and noisy labels. We evaluate our newly proposed models and related baselines in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise. The advanced modeling of the noisy labels substantially improves the performance up to 36% over methods without noise-handling and up to 9% over all other noise-handling baselines.

Question: How large is the corpus?
Answer: It contains 106,350 documents
Abstract: This paper explores a simple and efficient baseline for text classification. Our experiments show that our fast text classifier fastText is often on par with deep learning classifiers in terms of accuracy, and many orders of magnitude faster for training and evaluation. We can train fastText on more than one billion words in less than ten minutes using a standard multicore~CPU, and classify half a million sentences among~312K classes in less than a minute.

Introduction: Text classification is an important task in Natural Language Processing with many applications, such as web search, information retrieval, ranking and document classification BIBREF0 , BIBREF1 . Recently, models based on neural networks have become increasingly popular BIBREF2 , BIBREF3 , BIBREF4 . While these models achieve very good performance in practice, they tend to be relatively slow both at train and test time, limiting their use on very large datasets. Meanwhile, linear classifiers are often considered as strong baselines for text classification problems BIBREF5 , BIBREF6 , BIBREF7 . Despite their simplicity, they often obtain state-of-the-art performances if the right features are used BIBREF8 . They also have the potential to scale to very large corpus BIBREF9 . In this work, we explore ways to scale these baselines to very large corpus with a large output space, in the context of text classification. Inspired by the recent work in efficient word representation learning BIBREF10 , BIBREF11 , we show that linear models with a rank constraint and a fast loss approximation can train on a billion words within ten minutes, while achieving performance on par with the state-of-the-art. We evaluate the quality of our approach fastText on two different tasks, namely tag prediction and sentiment analysis.

Question: What is the performance of the models on the tasks?
Answer: Overall accuracy per model is: 5-gram (60.5), LSTM (68.9), TXL (68.7), GPT-2 (80.1)
Abstract: In this work, we present a simple yet better variant of Self-Critical Sequence Training. We make a simple change in the choice of baseline function in REINFORCE algorithm. The new baseline can bring better performance with no extra cost, compared to the greedy decoding baseline.

Introduction: Self-Critical Sequence Training(SCST), upon its release, has been a popular way to train sequence generation models. While originally proposed for image captioning task, SCST not only has become the new standard for training captioning models BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, but also has been applied to many other tasks, like video captioningBIBREF10, BIBREF11, BIBREF12, reading comprehensionBIBREF13, summarizationBIBREF14, BIBREF15, BIBREF16, BIBREF17, image paragraph generationBIBREF18, speech recognitionBIBREF19. SCST is used to optimize generated sequences over a non-differentiable objective, usually the evaluation metrics, for example, CIDEr for captioning, ROUGE for summarization. To optimize such objective, SCST adopts REINFORCE with baseline BIBREF20, where a “Self-Critical” baseline is used; specifically, the score of the greedy decoding output is used as the baseline. This is proved to be better than learned baseline function which is more commonly used in Reinforcement Learning literature. In this work, we present a different baseline choice which was first proposed in BIBREF21, to the best of our knowledge. With more elaboration in Sec. SECREF3, this baseline can be described as a variant of “Self-Critical”. This method is simple, but also faster and more effective compared to the greedy decoding baseline used in SCST.

Question: On what dataset is Aristo system trained?
Answer: Aristo Corpus
Regents 4th
Regents 8th
Regents `12th
ARC-Easy
ARC-challenge
Abstract: Contextual word embeddings (e.g. GPT, BERT, ELMo, etc.) have demonstrated state-of-the-art performance on various NLP tasks. Recent work with the multilingual version of BERT has shown that the model performs very well in zero-shot and zero-resource cross-lingual settings, where only labeled English data is used to finetune the model. We improve upon multilingual BERT's zero-resource cross-lingual performance via adversarial learning. We report the magnitude of the improvement on the multilingual MLDoc text classification and CoNLL 2002/2003 named entity recognition tasks. Furthermore, we show that language-adversarial training encourages BERT to align the embeddings of English documents and their translations, which may be the cause of the observed performance gains.

Introduction: Contextual word embeddings BIBREF0 , BIBREF1 , BIBREF2 have been successfully applied to various NLP tasks, including named entity recognition, document classification, and textual entailment. The multilingual version of BERT (which is trained on Wikipedia articles from 100 languages and equipped with a 110,000 shared wordpiece vocabulary) has also demonstrated the ability to perform `zero-resource' cross-lingual classification on the XNLI dataset BIBREF3 . Specifically, when multilingual BERT is finetuned for XNLI with English data alone, the model also gains the ability to handle the same task in other languages. We believe that this zero-resource transfer learning can be extended to other multilingual datasets. In this work, we explore BERT's zero-resource performance on the multilingual MLDoc classification and CoNLL 2002/2003 NER tasks. We find that the baseline zero-resource performance of BERT exceeds the results reported in other work, even though cross-lingual resources (e.g. parallel text, dictionaries, etc.) are not used during BERT pretraining or finetuning. We apply adversarial learning to further improve upon this baseline, achieving state-of-the-art zero-resource results. There are many recent approaches to zero-resource cross-lingual classification and NER, including adversarial learning BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , using a model pretrained on parallel text BIBREF8 , BIBREF9 , BIBREF10 and self-training BIBREF11 . Due to the newness of the subject matter, the definition of `zero-resource' varies somewhat from author to author. For the experiments that follow, `zero-resource' means that, during model training, we do not use labels from non-English data, nor do we use human or machine-generated parallel text. Only labeled English text and unlabeled non-English text are used during training, and hyperparameters are selected using English evaluation sets. Our contributions are the following:

Question: how do they measure discussion quality?
Answer: Measuring three aspects: argumentation, specificity and knowledge domain.
Abstract: Recently, neural models pretrained on a language modeling task, such as ELMo (Peters et al., 2017), OpenAI GPT (Radford et al., 2018), and BERT (Devlin et al., 2018), have achieved impressive results on various natural language processing tasks such as question-answering and natural language inference. In this paper, we describe a simple re-implementation of BERT for query-based passage re-ranking. Our system is the state of the art on the TREC-CAR dataset and the top entry in the leaderboard of the MS MARCO passage retrieval task, outperforming the previous state of the art by 27% (relative) in MRR@10. The code to reproduce our results is available at https://github.com/nyu-dl/dl4marco-bert

Introduction: We have seen rapid progress in machine reading compression in recent years with the introduction of large-scale datasets, such as SQuAD BIBREF3 , MS MARCO BIBREF4 , SearchQA BIBREF5 , TriviaQA BIBREF6 , and QUASAR-T BIBREF7 , and the broad adoption of neural models, such as BiDAF BIBREF8 , DrQA BIBREF9 , DocumentQA BIBREF10 , and QAnet BIBREF11 . The information retrieval (IR) community has also experienced a flourishing development of neural ranking models, such as DRMM BIBREF12 , KNRM BIBREF13 , Co-PACRR BIBREF14 , and DUET BIBREF15 . However, until recently, there were only a few large datasets for passage ranking, with the notable exception of the TREC-CAR BIBREF16 . This, at least in part, prevented the neural ranking models from being successful when compared to more classical IR techniques BIBREF17 . We argue that the same two ingredients that made possible much progress on the reading comprehension task are now available for passage ranking task. Namely, the MS MARCO passage ranking dataset, which contains one million queries from real users and their respective relevant passages annotated by humans, and BERT, a powerful general purpose natural language processing model. In this paper, we describe in detail how we have re-purposed BERT as a passage re-ranker and achieved state-of-the-art results on the MS MARCO passage re-ranking task.

Question: How big is slot filing dataset?
Answer: Dataset has 1737 train, 497 dev and 559 test sentences.
Abstract: Short-text classification, like all data science, struggles to achieve high performance using limited data. As a solution, a short sentence may be expanded with new and relevant feature words to form an artificially enlarged dataset, and add new features to testing data. This paper applies a novel approach to text expansion by generating new words directly for each input sentence, thus requiring no additional datasets or previous training. In this unsupervised approach, new keywords are formed within the hidden states of a pre-trained language model and then used to create extended pseudo documents. The word generation process was assessed by examining how well the predicted words matched to topics of the input sentence. It was found that this method could produce 3-10 relevant new words for each target topic, while generating just 1 word related to each non-target topic. Generated words were then added to short news headlines to create extended pseudo headlines. Experimental results have shown that models trained using the pseudo headlines can improve classification accuracy when limiting the number of training examples.

Introduction: The web has provided researchers with vast amounts of unlabeled text data, and enabled the development of increasingly sophisticated language models which can achieve state of the art performance despite having no task specific training BIBREF0, BIBREF1, BIBREF2. It is desirable to adapt these models for bespoke tasks such as short text classification. Short-text is nuanced, difficult to model statistically, and sparse in features, hindering traditional analysis BIBREF3. These difficulties become further compounded when training is limited, as is the case for many practical applications. This paper provides a method to expand short-text with additional keywords, generated using a pre-trained language model. The method takes advantage of general language understanding to suggest contextually relevant new words, without necessitating additional domain data. The method can form both derivatives of the input vocabulary, and entirely new words arising from contextualised word interactions and is ideally suited for applications where data volume is limited. figureBinary Classification of short headlines into 'WorldPost' or 'Crime' categories, shows improved performance with extended pseudo headlines when the training set is small. Using: Random forest classifier, 1000 test examples, 10-fold cross validation.

Question: What languages are represented in the dataset?
Answer: EN, JA, ES, AR, PT, KO, TH, FR, TR, RU, IT, DE, PL, NL, EL, SV, FA, VI, FI, CS, UK, HI, DA, HU, NO, RO, SR, LV, BG, UR, TA, MR, BN, IN, KN, ET, SL, GU, CY, ZH, CKB, IS, LT, ML, SI, IW, NE, KM, MY, TL, KA, BO
Abstract: One of the basic tasks of computational language documentation (CLD) is to identify word boundaries in an unsegmented phonemic stream. While several unsupervised monolingual word segmentation algorithms exist in the literature, they are challenged in real-world CLD settings by the small amount of available data. A possible remedy is to take advantage of glosses or translation in a foreign, well-resourced, language, which often exist for such data. In this paper, we explore and compare ways to exploit neural machine translation models to perform unsupervised boundary detection with bilingual information, notably introducing a new loss function for jointly learning alignment and segmentation. We experiment with an actual under-resourced language, Mboshi, and show that these techniques can effectively control the output segmentation length.

Introduction: All over the world, languages are disappearing at an unprecedented rate, fostering the need for specific tools aimed to aid field linguists to collect, transcribe, analyze, and annotate endangered language data (e.g. BIBREF0, BIBREF1). A remarkable effort in this direction has improved the data collection procedures and tools BIBREF2, BIBREF3, enabling to collect corpora for an increasing number of endangered languages (e.g. BIBREF4). One of the basic tasks of computational language documentation (CLD) is to identify word or morpheme boundaries in an unsegmented phonemic or orthographic stream. Several unsupervised monolingual word segmentation algorithms exist in the literature, based, for instance, on information-theoretic BIBREF5, BIBREF6 or nonparametric Bayesian techniques BIBREF7, BIBREF8. These techniques are, however, challenged in real-world settings by the small amount of available data. A possible remedy is to take advantage of glosses or translations in a foreign, well-resourced language (WL), which often exist for such data, hoping that the bilingual context will provide additional cues to guide the segmentation algorithm. Such techniques have already been explored, for instance, in BIBREF9, BIBREF10 in the context of improving statistical alignment and translation models; and in BIBREF11, BIBREF12, BIBREF13 using Attentional Neural Machine Translation (NMT) models. In these latter studies, word segmentation is obtained by post-processing attention matrices, taking attention information as a noisy proxy to word alignment BIBREF14. In this paper, we explore ways to exploit neural machine translation models to perform unsupervised boundary detection with bilingual information. Our main contribution is a new loss function for jointly learning alignment and segmentation in neural translation models, allowing us to better control the length of utterances. Our experiments with an actual under-resourced language (UL), Mboshi BIBREF17, show that this technique outperforms our bilingual segmentation baseline.

Question: How faster is training and decoding compared to former models?
Answer: Proposed vs best baseline:
Decoding: 8541 vs 8532 tokens/sec
Training: 8h vs 8h