Reviewerly/exHarmony huggingface dataset:
Understanding and Improving Knowledge Distillation for Quantization Aware Training of Large Transformer Encoders

Knowledge distillation (KD) has been a ubiquitous method for model compression to strengthen the capability of a lightweight model with the transferred knowledge from the teacher.In particular, KD has been employed in quantization-aware training (QAT) of Transformer encoders like BERT to improve the accuracy of the student model with the reducedprecision weight parameters.However, little is understood about which of the various KD approaches best fits the QAT of Transformers.In this work, we provide an in-depth analysis of the mechanism of KD on attention recovery of quantized large Transformers.In particular, we reveal that the previously adopted MSE loss on the attention score is insufficient for recovering the self-attention information.Therefore, we propose two KD methods; attention-map and attention-output losses.Furthermore, we explore the unification of both losses to address task-dependent preference between attentionmap and output losses.The experimental results on various Transformer encoder models demonstrate that the proposed KD methods achieve state-of-the-art accuracy for QAT with sub-2-bit weight quantization.
Explainable Question Answering based on Semantic Graph by Global Differentiable Learning and Dynamic Adaptive Reasoning

Multi-hop Question Answering is an agent task for testing the reasoning ability.With the development of pre-trained models, the implicit reasoning ability has been surprisingly improved and can even surpass human performance.However, the nature of the black box hinders the construction of explainable intelligent systems.Several researchers have explored explainable neural-symbolic reasoning methods based on question decomposition techniques.The undifferentiable symbolic operations and the error propagation in the reasoning process lead to poor performance.To alleviate it, we propose a simple yet effective Global Differentiable Learning strategy to explore optimal reasoning paths from the latent probability space so that the model learns to solve intermediate reasoning processes without expert annotations.We further design a Dynamic Adaptive Reasoner to enhance the generalization of unseen questions.Our method achieves 17% improvements in F1-score against BreakRC and shows better interpretability.We take a step forward in building interpretable reasoning methods.
Unbiased and Efficient Sampling of Dependency Trees

Most computational models of dependency syntax consist of distributions over spanning trees. However, the majority of dependency treebanks require that every valid dependency tree has a single edge coming out of the ROOT node, a constraint that is not part of the definition of spanning trees. For this reason all standard inference algorithms for spanning trees are sub-optimal for inference over dependency trees.Zmigrod et al (2021) proposed algorithms for sampling with and without replacement from the dependency tree distribution that incorporate the single-root constraint. In this paper we show that their fastest algorithm for sampling with replacement, Wilson-RC, is in fact producing biased samples and we provide two alternatives that are unbiased. Additionally, we propose two algorithms (one incremental, one parallel) that reduce the asymptotic runtime of algorithm for sampling k trees without replacement to O(kn^3). These algorithms are both asymptotically and practically more efficient.
Faithful Knowledge Graph Explanations in Commonsense Question Answering

Knowledge graphs are commonly used as sources of information in commonsense question answering, and can also be used to express explanations for the model's answer choice.A common way of incorporating facts from the graph is to encode them separately from the question, and then combine the two representations to select an answer.In this paper, we argue that highly faithful graph-based explanations cannot be extracted from existing models of this type.Such explanations will not include reasoning done by the transformer encoding the question, so will be incomplete.We confirm this theory with a novel proxy measure for faithfulness and propose two architecture changes to address the problem.Our findings suggest a path forward for developing architectures for faithful graph-based explanations.
Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training

Recently, knowledge-enhanced pre-trained language models (KEPLMs) improve contextaware representations via learning from structured relations in knowledge graphs, and/or linguistic knowledge from syntactic or dependency analysis.Unlike English, there is a lack of high-performing open-source Chinese KE-PLMs in the natural language processing (NLP) community to support various language understanding applications.In this paper, we revisit and advance the development of Chinese natural language understanding with a series of novel Chinese KEPLMs released in various parameter sizes, namely CKBERT (Chinese knowledge-enhanced BERT).Specifically, both relational and linguistic knowledge is effectively injected into CKBERT based on two novel pre-training tasks, i.e., linguistic-aware masked language modeling and contrastive multi-hop relation modeling.Based on the above two pre-training paradigms and our inhouse implemented TorchAccelerator, we have pre-trained base (110M), large (345M) and huge (1.3B) versions of CKBERT efficiently on GPU clusters.Experiments demonstrate that CKBERT outperforms strong baselines for Chinese over various benchmark NLP tasks and in terms of different model sizes.
Towards Pragmatic Production Strategies for Natural Language Generation Tasks

This position paper proposes a conceptual framework for the design of Natural Language Generation (NLG) systems that follow efficient and effective production strategies in order to achieve complex communicative goals.In this general framework, efficiency is characterised as the parsimonious regulation of production and comprehension costs while effectiveness is measured with respect to task-oriented and contextually grounded communicative goals.We provide concrete suggestions for the estimation of goals, costs, and utility via modern statistical methods, demonstrating applications of our framework to the classic pragmatic task of visually grounded referential games and to abstractive text summarisation, two popular generation tasks with real-world applications.In sum, we advocate for the development of NLG systems that learn to make pragmatic production decisions from experience, by reasoning about goals, costs, and utility in a human-like way.
An Adaptive Logical Rule Embedding Model for Inductive Reasoning over Temporal Knowledge Graphs

Temporal knowledge graphs (TKGs) extrapolation reasoning predicts future events based on historical information, which has great research significance and broad application value.Existing methods can be divided into embeddingbased methods and logical rule-based methods.Embedding-based methods rely on learned entity and relation embeddings to make predictions and thus lack interpretability.Logical rule-based methods bring scalability problems due to being limited by the learned logical rules.We combine the two methods to capture deep causal logic by learning rule embeddings, and propose an interpretable model for temporal knowledge graph reasoning called adaptive logical rule embedding model for inductive reasoning (ALRE-IR).ALRE-IR can adaptively extract and assess reasons contained in historical events, and make predictions based on causal logic.Furthermore, we propose a one-class augmented matching loss for optimization.When evaluated on ICEWS14, ICEWS0515 and ICEWS18 datasets, the performance of ALRE-IR outperforms other stateof-the-art baselines.The results also demonstrate that ALRE-IR still shows outstanding performance when transferred to related dataset with common relation vocabulary, indicating our proposed model has good zero-shot reasoning ability.
Weight Perturbation as Defense against Adversarial Word Substitutions

The existence and pervasiveness of textual adversarial examples have raised serious concerns to security-critical applications.Many methods have been developed to defend against adversarial attacks for neural natural language processing (NLP) models.Adversarial training is one of the most successful defense methods by adding some random or intentional perturbations to the original input texts and making the models robust to the perturbed examples.In this study, we explore the feasibility of improving the adversarial robustness of NLP models by performing perturbations in the parameter space rather than the input feature space.The weight perturbation helps to find a better solution (i.e., the values of weights) that minimizes the adversarial loss among other feasible solutions.We found that the weight perturbation can significantly improve the robustness of NLP models when it is combined with the perturbation in the input embedding space, yielding the highest accuracy on both clean and adversarial examples across different datasets.
X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization

Subhajit Chaudhury, Sarathkrishna Swaminathan, Chulaka Gunasekara, Maxwell Crouse, Srinivas Ravishankar, Daiki Kimura, Keerthiram Murugesan, Ramón Fernandez Astudillo, Tahira Naseem, Pavan Kapanipathi, Alexander Gray. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.
G3: Geolocation via Guidebook Grounding

We demonstrate how language can improve geolocation: the task of predicting the location where an image was taken.Here we study explicit knowledge from human-written guidebooks that describe the salient and classdiscriminative visual features humans use for geolocation.We propose the task of Geolocation via Guidebook Grounding that uses a dataset of StreetView images from a diverse set of locations and an associated textual guidebook for GeoGuessr, a popular interactive geolocation game.Our approach predicts a country for each image by attending over the clues automatically extracted from the guidebook.Supervising attention with country-level pseudo labels achieves the best performance.Our approach substantially outperforms a state-of-the-art image-only geolocation method, with an improvement of over 5% in Top-1 accuracy.Our dataset and code can be found at https://github.com/g-luo/geolocation_via_guidebook_grounding.
Disentangled Variational Topic Inference for Topic-Accurate Financial Report Generation

Automatic generating financial report from a set of news is important but challenging.The financial reports is composed of key points of the news and corresponding inferring and reasoning from specialists in financial domain with professional knowledge.The challenges lie in the effective learning of the extra knowledge that is not well presented in the news, and the misalignment between topic of input news and output knowledge in target reports.In this work, we introduce a disentangled variational topic inference approach to learn two latent variables for news and report, respectively.We use a publicly available dataset to evaluate the proposed approach.The results demonstrate its effectiveness of enhancing the language informativeness and the topic accuracy of the generated financial reports.
DialogConv: A Lightweight Fully Convolutional Network for Multi-view Response Selection

Current end-to-end retrieval-based dialogue systems are mainly based on Recurrent Neural Networks or Transformers with attention mechanisms.Although promising results have been achieved, these models often suffer from slow inference or huge number of parameters.In this paper, we propose a novel lightweight fully convolutional architecture, called DialogConv, for response selection.DialogConv is exclusively built on top of convolution to extract matching features of context and response.Dialogues are modeled in 3D views, where DialogConv performs convolution operations on embedding view, word view and utterance view to capture richer semantic information from multiple contextual views.On the four benchmark datasets, compared with state-of-the-art baselines, Di-alogConv is on average about 8.5× smaller in size, and 79.39× and 10.64× faster on CPU and GPU devices, respectively.At the same time, DialogConv achieves the competitive effectiveness of response selection.
Identifying Physical Object Use in Sentences

Commonsense knowledge about the typical functions of physical objects allows people to make inferences during sentence understanding.For example, we infer that "Sam enjoyed the book" means that Sam enjoyed reading the book, even though the action is implicit.Prior research has focused on learning the prototypical functions of physical objects in order to enable inferences about implicit actions.But many sentences refer to objects even when they are not used (e.g., "The book fell").We argue that NLP systems need to recognize whether an object is being used before inferring how the object is used.We define a new task called Object Use Classification that determines whether a physical object mentioned in a sentence was used or likely will be used.We introduce a new dataset for this task and present a classification model that exploits data augmentation methods and FrameNet when fine-tuning a pre-trained language model.We also show that object use classification combined with knowledge about the prototypical functions of objects has the potential to yield very good inferences about implicit and anticipated actions.
Improve Interpretability of Neural Networks via Sparse Contrastive Coding

Although explainable artificial intelligence (XAI) has achieved remarkable developments in recent years, there are few efforts have been devoted to the following problems, namely, i) how to develop an explainable method that could explain the black-box in a modelagnostic way? and ii) how to improve the performance and interpretability of the blackbox using such explanations instead of precollected important attributions?To explore the potential solution, we propose a modelagnostic explanation method termed as Sparse Contrastive Coding (SCC) and verify its effectiveness in text classification and natural language inference.In brief, SCC explains the feature attributions which characterize the importance of words based on the hidden states of each layer of the model.With such wordlevel explainability, SCC adaptively divides the input sentences into foregrounds and backgrounds in terms of task relevance.Through maximizing the similarity between the foregrounds and input sentences while minimizing the similarity between the backgrounds and input sentences, SSC employs a supervised contrastive learning loss to boost the interpretability and performance of the model.Extensive experiments show the superiority of our method over five state-of-the-art methods in terms of interpretability and classification measurements.The code is available at https: //pengxi.me.
Entity Retrieval from Multilingual Knowledge Graphs

Knowledge Graphs (KGs) are structured databases that capture real-world entities and their relationships.The task of entity retrieval from a KG aims at retrieving a ranked list of entities relevant to a given user query.While English-only entity retrieval has attracted considerable attention, user queries, as well as the information contained in the KG, may be represented in multiple-and possibly distinctlanguages.Furthermore, KG content may vary between languages due to different information sources and points of view.Recent advances in language representation have enabled natural ways of bridging gaps between languages.In this paper, we, therefore, propose to utilise language models (LMs) and diverse entity representations to enable truly multilingual entity retrieval.We propose two approaches: (i) an array of monolingual retrievers and (ii) a single multilingual retriever trained using queries and documents in multiple languages.We show that while our approach is on par with the significantly more complex state-of-the-art method for the English task, it can be successfully applied to virtually any language with an LM.Furthermore, it allows languages to benefit from one another, yielding significantly better performance, both for low-and high-resource languages.
Data Selection Curriculum for Neural Machine Translation

Neural Machine Translation (NMT) models are typically trained on heterogeneous data that are concatenated and randomly shuffled.However, not all of the training data are equally useful to the model.Curriculum training aims to present the data to the NMT models in a meaningful order.In this work, we introduce a two-stage training framework for NMT where we fine-tune a base NMT model on subsets of data, selected by both deterministic scoring using pre-trained methods and online scoring that considers prediction scores of the emerging NMT model.Through comprehensive experiments on six language pairs comprising lowand high-resource languages from WMT'21, we have shown that our curriculum strategies consistently demonstrate better quality (up to +2.2 BLEU improvement) and faster convergence (approximately 50% fewer updates).
A Survey of Recent Error Annotation Schemes for Automatically Generated Text

While automatically computing numerical scores remains the dominant paradigm in NLP system evaluation, error annotation and analysis is receiving increasing attention, with several error annotation schemes recently proposed for automatically generated text.However, there is little agreement about what error annotation schemes should look like, how many different types of errors should be distinguished and at what level of granularity.In this paper, our aim is to map out work on annotating errors in human and machine generated text, with a particular focus on error taxonomies.We describe our paper selection process, and survey the error annotation schemes reported in the papers, drawing out similarities and differences between them.Finally, we characterise the issues that would make it difficult to move from the current situation to a standardised error taxonomy for annotating errors in automatically generated text.
End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics

The advances in language-based Artificial Intelligence (AI) technologies applied to build educational applications can present AI for socialgood opportunities with a broader positive impact.Across many disciplines, enhancing the quality of mathematics education is crucial in building critical thinking and problem-solving skills at younger ages.Conversational AI systems have started maturing to a point where they could play a significant role in helping students learn fundamental math concepts.This work presents a task-oriented Spoken Dialogue System (SDS) built to support play-based learning of basic math concepts for early childhood education.The system has been evaluated via real-world deployments at school while the students are practicing early math concepts with multimodal interactions.We discuss our efforts to improve the SDS pipeline built for math learning, for which we explore utilizing Math-BERT (Shen et al., 2021b) representations for potential enhancement to the Natural Language Understanding (NLU) module.We perform an end-to-end evaluation using real-world deployment outputs from the Automatic Speech Recognition (ASR), Intent Recognition, and Dialogue Manager (DM) components to understand how error propagation affects the overall performance in real-world scenarios.
Q-CASA Invited Speaker Performance Evaluations of Noisy Approximate Quantum Fourier Arithmetic with Signed and Unsigned Integers

The Quantum Fourier Transform (QFT) grants competitive advantages for performing arithmetic operations on quantum computers, and presents a potential route towards a numerical quantum-computational paradigm. Qubit simulation allows us to better gauge what will be possible as the technology improves and identify strategies for early implementations on noisy quantum devices. In this presentation, I will review an implementation of QFT-based unsigned integer addition and multiplication, and present their performance evaluation using noise models based on IBM’s superconducting qubit architecture. We isolate the sensitivity of the component quantum circuits on both one-/two-qubit gate error rates, the number of the arithmetic operands’ superposed integer states, and level of circuit approximation in the QFT. I will then extend the discussion to how this approach may be implemented for signed quantum Fourier arithmetic and offer some preliminary results.
Saubhagya: An Online Food Donation Platform for Ending Hunger and Malnutrition in Sri Lanka

Hunger and malnutrition continue to be significant challenges in developing nations, including Sri Lanka. To address this issue, the research paper presents "Saubhagya," an online web application that provides a platform for social assistance. The platform allows individuals to donate food and groceries to needy organizations such as blind, deaf, orphanages, making it a user-friendly and effective solution. Users are required to register as food donators, needy people(organizations), partners and food collection agents. The system connects these user groups when necessary, ensuring a smooth and efficient process. One unique feature of "Saubhagya" is its live capability of tracking food collection and delivery using GPS and Google Maps. This feature ensures that food donations are delivered to the right organizations promptly, promoting transparency, accountability, and communication among users. The research paper aims to evaluate the effectiveness of "Saubhagya" in reducing hunger and malnutrition in Sri Lanka through user feedback and system performance metrics. If successful, the platform can be scaled to other developing nations facing similar challenges. This research demonstrates the potential of digital platforms in addressing social and environmental challenges. By leveraging technology, collective action can be harnessed to create positive social impact. "Saubhagya" represents a significant step forward in the fight against hunger and malnutrition, and it is hoped that it can inspire others to use technology to address pressing global issues.
Focused categorization power of ontologies: General framework and study on simple existential concept expressions

When reusing existing ontologies for publishing a dataset in RDF (or developing a new ontology), preference may be given to those providing extensive subcategorization for important classes (denoted as focus classes). The subcategories may consist not only of named classes but also of compound class expressions. We define the notion of focused categorization power of a given ontology, with respect to a focus class and a concept expression language, as the (estimated) weighted count of the categories that can be built from the ontology’s signature, conform to the language, and are subsumed by the focus class. For the sake of tractable initial experiments we then formulate a restricted concept expression language based on existential restrictions, and heuristically map it to syntactic patterns over ontology axioms (so-called FCE patterns). The characteristics of the chosen concept expression language and associated FCE patterns are investigated using three different empirical sources derived from ontology collections: first, the concept expression pattern frequency in class definitions; second, the occurrence of FCE patterns in the Tbox of ontologies; and last, for class expressions generated from the Tbox of ontologies (through the FCE patterns); their ‘meaningfulness’ was assessed by different groups of users, yielding a ‘quality ordering’ of the concept expression patterns. The complementary analyses are then compared and summarized. To allow for further experimentation, a web-based prototype was also implemented, which covers the whole process of ontology reuse from keyword-based ontology search through the FCP computation to the selection of ontologies and their enrichment with new concepts built from compound expressions.