mtybilly/Instruct2DS huggingface dataset:
A Three-Parameter Rank-Frequency Relation in Natural Languages
Authors: Masao Utiyama, Eiichiro Sumita Chenchen Ding
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: We present that, the rank-frequency relation in textual data follows $f ∝ r^-α(r+γ)^-β$, where $f$ is the token frequency and $r$ is the rank by frequency, with ($α$, $β$, $γ$) as parameters. The formulation is derived based on the empirical observation that $d^2 (x+y)/dx^2$ is a typical impulse function, where $(x,y)=(łog r, łog f)$. The formulation is the power law when $β=0$ and the Zipf--Mandelbrot law when $α=0$. We illustrate that $α$ is related to the analytic features of syntax and $β+γ$ to those of morphology in natural languages from an investigation of multilingual corpora.
Dice Loss for Data-imbalanced NLP Tasks
Authors: Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, Jiwei Li Xiaoya Li
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Many NLP tasks such as tagging and machine reading comprehension are faced with the severe data imbalance issue: negative examples significantly outnumber positive examples, and the huge number of easy-negative examples overwhelms the training. The most commonly used cross entropy (CE) criteria is actually an accuracy-oriented objective, and thus creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive examples. In this paper, we propose to use dice loss in replacement of the standard cross-entropy objective for data-imbalanced NLP tasks. Dice loss is based on the Sørensen--Dice coefficient or Tversky index , which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To further alleviate the dominating influence from easy-negative examples in training, we propose to associate training examples with dynamically adjusted weights to deemphasize easy-negative examples. Theoretical analysis shows that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training. With the proposed training objective, we observe significant performance boost on a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task; SOTA results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named entity recognition task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification.
Emergence of Syntax Needs Minimal Supervision
Authors: Kata Gábor Raphaël Bailly
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: This paper is a theoretical contribution to the debate on the learnability of syntax from a corpus without explicit syntax-specific guidance. Our approach originates in the observable structure of a corpus, which we use to define and isolate grammaticality (syntactic information) and meaning/pragmatics information. We describe the formal characteristics of an autonomous syntax and show that it becomes possible to search for syntax-based lexical categories with a simple optimization process, without any prior hypothesis on the form of the model.
Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese
Authors: Takumi Ito, Jun Suzuki, Kentaro Inui Tatsuki Kuribayashi
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: We examine a methodology using neural language models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties existing methods face, such as the propagation of preprocessor errors in count-based methods. In this study, we explore whether the LM-based method is valid for analyzing the word order. As a case study, this study focuses on Japanese due to its complex and flexible word order. To validate the LM-based method, we test (i) parallels between LMs and human word order preference, and (ii) consistency of the results obtained using the LM-based method with previous linguistic studies. Through our experiments, we tentatively conclude that LMs display sufficient word order knowledge for usage as an analysis tool. Finally, using the LM-based method, we demonstrate the relationship between the canonical word order and topicalization, which had yet to be analyzed by large-scale experiments.
GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media
Authors: Cheng-Te Li Yi-Ju Lu
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: This paper solves the fake news detection problem under a more realistic scenario on social media. Given the source short-text tweet and the corresponding sequence of retweet users without text comments, we aim at predicting whether the source tweet is fake or not, and generating explanation by highlighting the evidences on suspicious retweeters and the words they concern. We develop a novel neural network-based model, Graph-aware Co-Attention Networks (GCAN), to achieve the goal. Extensive experiments conducted on real tweet datasets exhibit that GCAN can significantly outperform state-of-the-art methods by 16% in accuracy on average. In addition, the case studies also show that GCAN can produce reasonable explanations.
Integrating Semantic and Structural Information with Graph Convolutional Network for Controversy Detection
Authors: Juan Cao, Qiang Sheng, Junbo Guo, Ziang Wang Lei Zhong
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Identifying controversial posts on social media is a fundamental task for mining public sentiment, assessing the influence of events, and alleviating the polarized views. However, existing methods fail to 1) effectively incorporate the semantic information from content-related posts; 2) preserve the structural information for reply relationship modeling; 3) properly handle posts from topics dissimilar to those in the training set. To overcome the first two limitations, we propose Topic-Post-Comment Graph Convolutional Network (TPC-GCN), which integrates the information from the graph structure and content of topics, posts, and comments for post-level controversy detection. As to the third limitation, we extend our model to Disentangled TPC-GCN (DTPC-GCN), to disentangle topic-related and topic-unrelated features and then fuse dynamically. Extensive experiments on two real-world datasets demonstrate that our models outperform existing methods. Analysis of the results and cases proves that our models can integrate both semantic and structural information with significant generalizability.
Predicting the Topical Stance and Political Leaning of Media using Tweets
Authors: Kareem Darwish, Atanas Atanasov, Preslav Nakov Peter Stefanov
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Discovering the stances of media outlets and influential people on current, debatable topics is important for social statisticians and policy makers. Many supervised solutions exist for determining viewpoints, but manually annotating training data is costly. In this paper, we propose a cascaded method that uses unsupervised learning to ascertain the stance of Twitter users with respect to a polarizing topic by leveraging their retweet behavior; then, it uses supervised learning based on user labels to characterize both the general political leaning of online media and of popular Twitter users, as well as their stance with respect to the target polarizing topic. We evaluate the model by comparing its predictions to gold labels from the Media Bias/Fact Check website, achieving 82.6% accuracy.
Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora
Authors: Ganesh Jawahar, Djamé Seddah, Yoav Goldberg Hila Gonen
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: The problem of comparing two bodies of text and searching for words that differ in their usage between them arises often in digital humanities and computational social science. This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large. However, these methods often require extensive filtering of the vocabulary to perform well, and - as we show in this work - result in unstable, and hence less reliable, results. We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word. The method is simple, interpretable and stable. We demonstrate its effectiveness in 9 different setups, considering different corpus splitting criteria (age, gender and profession of tweet authors, time of tweet) and different languages (English, French and Hebrew).
CDL: Curriculum Dual Learning for Emotion-Controllable Response Generation
Authors: Yang Feng Lei Shen
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Emotion-controllable response generation is an attractive and valuable task that aims to make open-domain conversations more empathetic and engaging. Existing methods mainly enhance the emotion expression by adding regularization terms to standard cross-entropy loss and thus influence the training process. However, due to the lack of further consideration of content consistency, the common problem of response generation tasks, safe response, is intensified. Besides, query emotions that can help model the relationship between query and response are simply ignored in previous models, which would further hurt the coherence. To alleviate these problems, we propose a novel framework named Curriculum Dual Learning (CDL) which extends the emotion-controllable response generation to a dual task to generate emotional responses and emotional queries alternatively. CDL utilizes two rewards focusing on emotion and content to improve the duality. Additionally, it applies curriculum learning to gradually generate high-quality responses based on the difficulties of expressing various emotions. Experimental results show that CDL significantly outperforms the baselines in terms of coherence, diversity, and relation to emotion factors.
Efficient Dialogue State Tracking by Selectively Overwriting Memory
Authors: Sohee Yang, Gyuwan Kim, Sang-Woo Lee Sungdong Kim
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Recent works in dialogue state tracking (DST) focus on an open vocabulary-based setting to resolve scalability and generalization issues of the predefined ontology-based approaches. However, they are inefficient in that they predict the dialogue state at every turn from scratch. Here, we consider dialogue state as an explicit fixed-sized memory and propose a selectively overwriting mechanism for more efficient DST. This mechanism consists of two steps: (1) predicting state operation on each of the memory slots, and (2) overwriting the memory with new values, of which only a few are generated according to the predicted state operations. Our method decomposes DST into two sub-tasks and guides the decoder to focus only on one of the tasks, thus reducing the burden of the decoder. This enhances the effectiveness of training and DST performance. Our SOM-DST (Selectively Overwriting Memory for Dialogue State Tracking) model achieves state-of-the-art joint goal accuracy with 51.72% in MultiWOZ 2.0 and 53.01% in MultiWOZ 2.1 in an open vocabulary-based DST setting. In addition, we analyze the accuracy gaps between the current and the ground truth-given situations and suggest that it is a promising direction to improve state operation prediction to boost the DST performance.
End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2
Authors: Jeong-Gwan Lee, Youngsoo Jang, Kee-Eung Kim Donghoon Ham
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: The goal-oriented dialogue system needs to be optimized for tracking the dialogue flow and carrying out an effective conversation under various situations to meet the user goal. The traditional approach to build such a dialogue system is to take a pipelined modular architecture, where its modules are optimized individually. However, such an optimization scheme does not necessarily yield the overall performance improvement of the whole system. On the other hand, end-to-end dialogue systems with monolithic neural architecture are often trained only with input-output utterances, without taking into account the entire annotations available in the corpus. This scheme makes it difficult for goal-oriented dialogues where the system needs to integrate with external systems or to provide interpretable information about why the system generated a particular response. In this paper, we present an end-to-end neural architecture for dialogue systems that addresses both challenges above. In the human evaluation, our dialogue system achieved the success rate of 68.32%, the language understanding score of 4.149, and the response appropriateness score of 4.287, which ranked the system at the top position in the end-to-end multi-domain dialogue system task in the 8th dialogue systems technology challenge (DSTC8).
Evaluating Dialogue Generation Systems via Response Selection
Authors: Reina Akama, Hiroki Ouchi, Jun Suzuki, Kentaro Inui Shiki Sato
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Existing automatic evaluation metrics for open-domain dialogue response generation systems correlate poorly with human evaluation. We focus on evaluating response generation systems via response selection. To evaluate systems properly via response selection, we propose a method to construct response selection test sets with well-chosen false candidates. Specifically, we propose to construct test sets filtering out some types of false candidates: (i) those unrelated to the ground-truth response and (ii) those acceptable as appropriate responses. Through experiments, we demonstrate that evaluating systems via response selection with the test set developed by our method correlates more strongly with human evaluation, compared with widely used automatic evaluation metrics such as BLEU.
Gated Convolutional Bidirectional Attention-based Model for Off-topic Spoken Response Detection
Authors: Ruobing Li, Hui Lin Yefei Zha
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Off-topic spoken response detection, the task aiming at predicting whether a response is off-topic for the corresponding prompt, is important for an automated speaking assessment system. In many real-world educational applications, off-topic spoken response detectors are required to achieve high recall for off-topic responses not only on seen prompts but also on prompts that are unseen during training. In this paper, we propose a novel approach for off-topic spoken response detection with high off-topic recall on both seen and unseen prompts. We introduce a new model, Gated Convolutional Bidirectional Attention-based Model (GCBiA), which applies bi-attention mechanism and convolutions to extract topic words of prompts and key-phrases of responses, and introduces gated unit and residual connections between major layers to better represent the relevance of responses and prompts. Moreover, a new negative sampling method is proposed to augment training data. Experiment results demonstrate that our novel approach can achieve significant improvements in detecting off-topic responses with extremely high on-topic recall, for both seen and unseen prompts.
Learning Low-Resource End-To-End Goal-Oriented Dialog for Fast and Reliable System Deployment
Authors: Hangyu Li, Chengguang Tang, Yongbin Li, Jian Sun, Xiaodan Zhu Yinpei Dai
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Existing end-to-end dialog systems perform less effectively when data is scarce. To obtain an acceptable success in real-life online services with only a handful of training examples, both fast adaptability and reliable performance are highly desirable for dialog systems. In this paper, we propose the Meta-Dialog System (MDS), which combines the advantages of both meta-learning approaches and human-machine collaboration. We evaluate our methods on a new extended-bAbI dataset and a transformed MultiWOZ dataset for low-resource goal-oriented dialog learning. Experimental results show that MDS significantly outperforms non-meta-learning baselines and can achieve more than 90% per-turn accuracies with only 10 dialogs on the extended-bAbI dataset.
Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge
Authors: Yuanmeng Yan, Weiran Xu Keqing He
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Neural-based context-aware models for slot tagging have achieved state-of-the-art performance. However, the presence of OOV(out-of-vocab) words significantly degrades the performance of neural-based models, especially in a few-shot scenario. In this paper, we propose a novel knowledge-enhanced slot tagging model to integrate contextual representation of input text and the large-scale lexical background knowledge. Besides, we use multi-level graph attention to explicitly model lexical relations. The experiments show that our proposed knowledge integration mechanism achieves consistent improvements across settings with different sizes of training data on two public benchmark datasets.
Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition
Authors: Runze Liang, Minlie Huang Ryuichi Takanobu
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Many studies have applied reinforcement learning to train a dialog policy and show great promise these years. One common approach is to employ a user simulator to obtain a large number of simulated user experiences for reinforcement learning algorithms. However, modeling a realistic user simulator is challenging. A rule-based simulator requires heavy domain expertise for complex tasks, and a data-driven simulator requires considerable data and it is even unclear how to evaluate a simulator. To avoid explicitly building a user simulator beforehand, we propose Multi-Agent Dialog Policy Learning, which regards both the system and the user as the dialog agents. Two agents interact with each other and are jointly learned simultaneously. The method uses the actor-critic framework to facilitate pretraining and improve scalability. We also propose Hybrid Value Network for the role-aware reward decomposition to integrate role-specific domain knowledge of each agent in the task-oriented dialog. Results show that our method can successfully build a system policy and a user policy simultaneously, and two agents can achieve a high task success rate through conversational interaction.
Paraphrase Augmented Task-Oriented Dialog Generation
Authors: Yichi Zhang, Zhijian Ou, Zhou Yu Silin Gao
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Neural generative models have achieved promising performance on dialog generation tasks if given a huge data set. However, the lack of high-quality dialog data and the expensive data annotation process greatly limit their application in real world settings. We propose a paraphrase augmented response generation (PARG) framework that jointly trains a paraphrase model and a response generation model to improve the dialog generation performance. We also design a method to automatically construct paraphrase training data set based on dialog state and dialog act labels. PARG is applicable to various dialog generation models, such as TSCP (Lei et al., 2018) and DAMD (Zhang et al., 2019). Experimental results show that the proposed framework improves these state-of-the-art dialog models further on CamRest676 and MultiWOZ. PARG also outperforms other data augmentation methods significantly in dialog generation tasks, especially under low resource settings.
Response-Anticipated Memory for On-Demand Knowledge Integration in Response Generation
Authors: Wei Bi, Dongkyu Lee, Lanqing Xue, Yiping Song, Xiaojiang Liu, Nevin L. Zhang Zhiliang Tian
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Neural conversation models are known to generate appropriate but non-informative responses in general. A scenario where informativeness can be significantly enhanced is Conversing by Reading (CbR), where conversations take place with respect to a given external document. In previous work, the external document is utilized by (1) creating a context-aware document memory that integrates information from the document and the conversational context, and then (2) generating responses referring to the memory. In this paper, we propose to create the document memory with some anticipated responses in mind. This is achieved using a teacher-student framework. The teacher is given the external document, the context, and the ground-truth response, and learns how to build a response-aware document memory from three sources of information. The student learns to construct a response-anticipated document memory from the first two sources, and teacher`s insight on memory creation. Empirical results show that our model outperforms the previous state-of-the-art for the CbR task.
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation
Authors: Jianzhong Qi, Yu Sun, Rui Zhang Xinting Huang
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Dialogue policy optimization often obtains feedback until task completion in task-oriented dialogue systems. This is insufficient for training intermediate dialogue turns since supervision signals (or rewards) are only provided at the end of dialogues. To address this issue, reward learning has been introduced to learn from state-action pairs of an optimal policy to provide turn-by-turn rewards. This approach requires complete state-action annotations of human-to-human dialogues (i.e., expert demonstrations), which is labor intensive. To overcome this limitation, we propose a novel reward learning approach for semi-supervised policy learning. The proposed approach learns a dynamics model as the reward function which models dialogue progress (i.e., state-action sequences) based on expert demonstrations, either with or without annotations. The dynamics model computes rewards by predicting whether the dialogue progress is consistent with expert demonstrations. We further propose to learn action embeddings for a better generalization of the reward function. The proposed approach outperforms competitive policy learning baselines on MultiWOZ, a benchmark multi-domain dataset.
Towards Unsupervised Language Understanding and Generation by Joint Dual Learning
Authors: Chao-Wei Huang, Yun-Nung Chen Shang-Yu Su
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: In modular dialogue systems, natural language understanding (NLU) and natural language generation (NLG) are two critical components, where NLU extracts the semantics from the given texts and NLG is to construct corresponding natural language sentences based on the input semantic representations. However, the dual property between understanding and generation has been rarely explored. The prior work is the first attempt that utilized the duality between NLU and NLG to improve the performance via a dual supervised learning framework. However, the prior work still learned both components in a supervised manner; instead, this paper introduces a general learning framework to effectively exploit such duality, providing flexibility of incorporating both supervised and unsupervised learning algorithms to train language understanding and generation models in a joint fashion. The benchmark experiments demonstrate that the proposed approach is capable of boosting the performance of both NLU and NLG. The source code is available at: r̆lhttps://github.com/MiuLab/DuaLUG.
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Authors: Maxine Eskenazi Shikib Mehri
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for evaluating dialog models. To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces interpretable measures for several desirable properties of dialog.
Explicit Semantic Decomposition for Definition Generation
Authors: Yu Bao, Shujian Huang, Xinyu Dai, Jiajun Chen Jiahuan Li
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Definition generation, which aims to automatically generate dictionary definitions for words, has recently been proposed to assist the construction of dictionaries and help people understand unfamiliar texts. However, previous works hardly consider explicitly modeling the “components” of definitions, leading to under-specific generation results. In this paper, we propose ESD, namely Explicit Semantic Decomposition for definition Generation, which explicitly decomposes the meaning of words into semantic components, and models them with discrete latent variables for definition generation. Experimental results show that achieves top results on WordNet and Oxford benchmarks, outperforming strong previous baselines.
Improved Natural Language Generation via Loss Truncation
Authors: Tatsunori B. Hashimoto Daniel Kang
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Neural language models are usually trained to match the distributional properties of large-scale corpora by minimizing the log loss. While straightforward to optimize, this approach forces the model to reproduce all variations in the dataset, including noisy and invalid references (e.g., misannotations and hallucinated facts). Even a small fraction of noisy data can degrade the performance of log loss. As an alternative, prior work has shown that minimizing the distinguishability of generated samples is a principled and robust loss that can handle invalid references. However, distinguishability has not been used in practice due to challenges in optimization and estimation. We propose loss truncation: a simple and scalable procedure which adaptively removes high log loss examples as a way to optimize for distinguishability. Empirically, we demonstrate that loss truncation outperforms existing baselines on distinguishability on a summarization task. Furthermore, we show that samples generated by the loss truncation model have factual accuracy ratings that exceed those of baselines and match human references.
Line Graph Enhanced AMR-to-Text Generation with Mix-Order Graph Attention Networks
Authors: Lu Chen, Zhi Chen, Ruisheng Cao, Su Zhu, Kai Yu Yanbin Zhao
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Efficient structure encoding for graphs with labeled edges is an important yet challenging point in many graph-based models. This work focuses on AMR-to-text generation -- A graph-to-sequence task aiming to recover natural language from Abstract Meaning Representations (AMR). Existing graph-to-sequence approaches generally utilize graph neural networks as their encoders, which have two limitations: 1) The message propagation process in AMR graphs is only guided by the first-order adjacency information. 2) The relationships between labeled edges are not fully considered. In this work, we propose a novel graph encoding framework which can effectively explore the edge relations. We also adopt graph attention networks with higher-order neighborhood information to encode the rich structure in AMR graphs. Experiment results show that our approach obtains new state-of-the-art performance on English AMR benchmark datasets. The ablation analyses also demonstrate that both edge relations and higher-order information are beneficial to graph-to-sequence modeling.
Rigid Formats Controlled Text Generation
Authors: Haisong Zhang, Xiaojiang Liu, Shuming Shi Piji Li
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Neural text generation has made tremendous progress in various tasks. One common characteristic of most of the tasks is that the texts are not restricted to some rigid formats when generating. However, we may confront some special text paradigms such as Lyrics (assume the music score is given), Sonnet, SongCi (classical Chinese poetry of the Song dynasty), etc. The typical characteristics of these texts are in three folds: (1) They must comply fully with the rigid predefined formats. (2) They must obey some rhyming schemes. (3) Although they are restricted to some formats, the sentence integrity must be guaranteed. To the best of our knowledge, text generation based on the predefined rigid formats has not been well investigated. Therefore, we propose a simple and elegant framework named SongNet to tackle this problem. The backbone of the framework is a Transformer-based auto-regressive language model. Sets of symbols are tailor-designed to improve the modeling performance especially on format, rhyme, and sentence integrity. We improve the attention mechanism to impel the model to capture some future information on the format. A pre-training and fine-tuning framework is designed to further improve the generation quality. Extensive experiments conducted on two collected corpora demonstrate that our proposed framework generates significantly better results in terms of both automatic metrics and the human evaluation.
Syn-QG: Syntactic and Shallow Semantic Rules for Question Generation
Authors: Christopher D. Manning Kaustubh Dhole
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Question Generation (QG) is fundamentally a simple syntactic transformation; however, many aspects of semantics influence what questions are good to form. We implement this observation by developing Syn-QG, a set of transparent syntactic rules leveraging universal dependencies, shallow semantic parsing, lexical resources, and custom rules which transform declarative sentences into question-answer pairs. We utilize PropBank argument descriptions and VerbNet state predicates to incorporate shallow semantic content, which helps generate questions of a descriptive nature and produce inferential and semantically richer questions than existing systems. In order to improve syntactic fluency and eliminate grammatically incorrect questions, we employ back-translation over the output of these syntactic rules. A set of crowd-sourced evaluations shows that our system can generate a larger number of highly grammatical and relevant questions than previous QG systems and that back-translation drastically improves grammaticality at a slight cost of generating irrelevant questions.
An Online Semantic-enhanced Dirichlet Model for Short Text Stream Clustering
Authors: Junming Shao, Salah Uddin, Wazir Ali Jay Kumar
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Clustering short text streams is a challenging task due to its unique properties: infinite length, sparse data representation and cluster evolution. Existing approaches often exploit short text streams in a batch way. However, determine the optimal batch size is usually a difficult task since we have no priori knowledge when the topics evolve. In addition, traditional independent word representation in graphical model tends to cause “term ambiguity” problem in short text clustering. Therefore, in this paper, we propose an Online Semantic-enhanced Dirichlet Model for short sext stream clustering, called OSDM, which integrates the word-occurance semantic information (i.e., context) into a new graphical model and clusters each arriving short text automatically in an online way. Extensive results have demonstrated that OSDM has better performance compared to many state-of-the-art algorithms on both synthetic and real-world data sets.
Generative Semantic Hashing Enhanced via Boltzmann Machines
Authors: Qinliang Su, Dinghan Shen, Changyou Chen Lin Zheng
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Generative semantic hashing is a promising technique for large-scale information retrieval thanks to its fast retrieval speed and small memory footprint. For the tractability of training, existing generative-hashing methods mostly assume a factorized form for the posterior distribution, enforcing independence among the bits of hash codes. From the perspectives of both model representation and code space size, independence is always not the best assumption. In this paper, to introduce correlations among the bits of hash codes, we propose to employ the distribution of Boltzmann machine as the variational posterior. To address the intractability issue of training, we first develop an approximate method to reparameterize the distribution of a Boltzmann machine by augmenting it as a hierarchical concatenation of a Gaussian-like distribution and a Bernoulli distribution. Based on that, an asymptotically-exact lower bound is further derived for the evidence lower bound (ELBO). With these novel techniques, the entire model can be optimized efficiently. Extensive experimental results demonstrate that by effectively modeling correlations among different bits within a hash code, our model can achieve significant performance gains.
Interactive Construction of User-Centric Dictionary for Text Analytics
Authors: Issei Yoshida, Hiroshi Kanayama, Tetsuya Nasukawa Ryosuke Kohita
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: We propose a methodology to construct a term dictionary for text analytics through an interactive process between a human and a machine, which helps the creation of flexible dictionaries with precise granularity required in typical text analysis. This paper introduces the first formulation of interactive dictionary construction to address this issue. To optimize the interaction, we propose a new algorithm that effectively captures an analyst`s intention starting from only a small number of sample terms. Along with the algorithm, we also design an automatic evaluation framework that provides a systematic assessment of any interactive method for the dictionary creation task. Experiments using real scenario based corpora and dictionaries show that our algorithm outperforms baseline methods, and works even with a small number of interactions.
Tree-Structured Neural Topic Model
Authors: Junichiro Mori, Danushka Bollegala, Ichiro Sakata Masaru Isonuma
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: This paper presents a tree-structured neural topic model, which has a topic distribution over a tree with an infinite number of branches. Our model parameterizes an unbounded ancestral and fraternal topic distribution by applying doubly-recurrent neural networks. With the help of autoencoding variational Bayes, our model improves data scalability and achieves competitive performance when inducing latent topics and tree structures, as compared to a prior tree-structured topic model (Blei et al., 2010). This work extends the tree-structured topic model such that it can be incorporated with neural models for downstream tasks.
Unsupervised FAQ Retrieval with Question Generation and BERT
Authors: Boaz Carmeli, Haggai Roitman, David Konopnicki Yosi Mass
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: We focus on the task of Frequently Asked Questions (FAQ) retrieval. A given user query can be matched against the questions and/or the answers in the FAQ. We present a fully unsupervised method that exploits the FAQ pairs to train two BERT models. The two models match user queries to FAQ answers and questions, respectively. We alleviate the missing labeled data of the latter by automatically generating high-quality question paraphrases. We show that our model is on par and even outperforms supervised models on existing datasets.
“The Boating Store Had Its Best Sail Ever”: Pronunciation-attentive Contextualized Pun Recognition
Authors: Jyun-Yu Jiang, Jieyu Zhao, Kai-Wei Chang, Wei Wang Yichao Zhou
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Humor plays an important role in human languages and it is essential to model humor when building intelligence systems. Among different forms of humor, puns perform wordplay for humorous effects by employing words with double entendre and high phonetic similarity. However, identifying and modeling puns are challenging as puns usually involved implicit semantic or phonological tricks. In this paper, we propose Pronunciation-attentive Contextualized Pun Recognition (PCPR) to perceive human humor, detect if a sentence contains puns and locate them in the sentence. PCPR derives contextualized representation for each word in a sentence by capturing the association between the surrounding context and its corresponding phonetic symbols. Extensive experiments are conducted on two benchmark datasets. Results demonstrate that the proposed approach significantly outperforms the state-of-the-art methods in pun detection and location tasks. In-depth analyses verify the effectiveness and robustness of PCPR.
Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning
Authors: Yoonhyung Lee, Seunghyun Yoon, Kyomin Jung Joongbo Shin
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Even though BERT has achieved successful performance improvements in various supervised learning tasks, BERT is still limited by repetitive inferences on unsupervised tasks for the computation of contextual language representations. To resolve this limitation, we propose a novel deep bidirectional language model called a Transformer-based Text Autoencoder (T-TA). The T-TA computes contextual language representations without repetition and displays the benefits of a deep bidirectional architecture, such as that of BERT. In computation time experiments in a CPU environment, the proposed T-TA performs over six times faster than the BERT-like model on a reranking task and twelve times faster on a semantic similarity task. Furthermore, the T-TA shows competitive or even better accuracies than those of BERT on the above tasks. Code is available at r̆lhttps://github.com/joongbo/tta.
Fine-grained Interest Matching for Neural News Recommendation
Authors: Fangzhao Wu, Zheng Liu, Xing Xie Heyuan Wang
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Personalized news recommendation is a critical technology to improve users' online news reading experience. The core of news recommendation is accurate matching between user`s interests and candidate news. The same user usually has diverse interests that are reflected in different news she has browsed. Meanwhile, important semantic features of news are implied in text segments of different granularities. Existing studies generally represent each user as a single vector and then match the candidate news vector, which may lose fine-grained information for recommendation. In this paper, we propose FIM, a Fine-grained Interest Matching method for neural news recommendation. Instead of aggregating user`s all historical browsed news into a unified vector, we hierarchically construct multi-level representations for each news via stacked dilated convolutions. Then we perform fine-grained matching between segment pairs of each browsed news and the candidate news at each semantic level. High-order salient signals are then identified by resembling the hierarchy of image recognition for final click prediction. Extensive experiments on a real-world dataset from MSN news validate the effectiveness of our model on news recommendation.
Interpretable Operational Risk Classification with Semi-Supervised Variational Autoencoder
Authors: Shengming Zhang, Yi Yang Fan Zhou
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Operational risk management is one of the biggest challenges nowadays faced by financial institutions. There are several major challenges of building a text classification system for automatic operational risk prediction, including imbalanced labeled/unlabeled data and lacking interpretability. To tackle these challenges, we present a semi-supervised text classification framework that integrates multi-head attention mechanism with Semi-supervised variational inference for Operational Risk Classification (SemiORC). We empirically evaluate the framework on a real-world dataset. The results demonstrate that our method can better utilize unlabeled data and learn visually interpretable document representations. SemiORC also outperforms other baseline methods on operational risk classification.
Interpreting Twitter User Geolocation
Authors: Tianliang Wang, Fan Zhou, Goce Trajcevski, Kunpeng Zhang, Yi Yang Ting Zhong
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Identifying user geolocation in online social networks is an essential task in many location-based applications. Existing methods rely on the similarity of text and network structure, however, they suffer from a lack of interpretability on the corresponding results, which is crucial for understanding model behavior. In this work, we adopt influence functions to interpret the behavior of GNN-based models by identifying the importance of training users when predicting the locations of the testing users. This methodology helps with providing meaningful explanations on prediction results. Furthermore, it also initiates an attempt to uncover the so-called “black-box” GNN-based models by investigating the effect of individual nodes.
Modeling Code-Switch Languages Using Bilingual Parallel Corpus
Authors: Haizhou Li Grandee Lee
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Language modeling is the technique to estimate the probability of a sequence of words. A bilingual language model is expected to model the sequential dependency for words across languages, which is difficult due to the inherent lack of suitable training data as well as diverse syntactic structure across languages. We propose a bilingual attention language model (BALM) that simultaneously performs language modeling objective with a quasi-translation objective to model both the monolingual as well as the cross-lingual sequential dependency. The attention mechanism learns the bilingual context from a parallel corpus. BALM achieves state-of-the-art performance on the SEAME code-switch database by reducing the perplexity of 20.5% over the best-reported result. We also apply BALM in bilingual lexicon induction, and language normalization tasks to validate the idea.
SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check
Authors: Weidi Xu, Kunlong Chen, Shaohua Jiang, Feng Wang, Taifeng Wang, Wei Chu, Yuan Qi Xingyi Cheng
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Chinese Spelling Check (CSC) is a task to detect and correct spelling errors in Chinese natural language. Existing methods have made attempts to incorporate the similarity knowledge between Chinese characters. However, they take the similarity knowledge as either an external input resource or just heuristic rules. This paper proposes to incorporate phonological and visual similarity knowledge into language models for CSC via a specialized graph convolutional network (SpellGCN). The model builds a graph over the characters, and SpellGCN is learned to map this graph into a set of inter-dependent character classifiers. These classifiers are applied to the representations extracted by another network, such as BERT, enabling the whole network to be end-to-end trainable. Experiments are conducted on three human-annotated datasets. Our method achieves superior performance against previous models by a large margin.
Spelling Error Correction with Soft-Masked BERT
Authors: Haoran Huang, Jicong Liu, Hang Li Shaohua Zhang
Published in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Main Conference Track), 2020

Abstract: Spelling error correction is an important yet challenging task because a satisfactory solution of it essentially needs human-level language understanding ability. Without loss of generality we consider Chinese spelling error correction (CSC) in this paper. A state-of-the-art method for the task selects a character from a list of candidates for correction (including non-correction) at each position of the sentence on the basis of BERT, the language representation model. The accuracy of the method can be sub-optimal, however, because BERT does not have sufficient capability to detect whether there is an error at each position, apparently due to the way of pre-training it using mask language modeling. In this work, we propose a novel neural architecture to address the aforementioned issue, which consists of a network for error detection and a network for error correction based on BERT, with the former being connected to the latter with what we call soft-masking technique. Our method of using \textquoteleftSoft-Masked BERT' is general, and it may be employed in other language detection-correction problems. Experimental results on two datasets, including one large dataset which we create and plan to release, demonstrate that the performance of our proposed method is significantly better than the baselines including the one solely based on BERT.