Reviewerly/exHarmony huggingface dataset:
DISCOSQA: A Knowledge Base Question Answering System for Space Debris based on Program Induction

Space agencies execute complex satellite operations that need to be supported by the technical knowledge contained in their extensive information systems.Knowledge bases (KB) are an effective way of storing and accessing such information at scale.In this work we present a system, developed for the European Space Agency (ESA), that can answer complex natural language queries, to support engineers in accessing the information contained in a KB that models the orbital space debris environment.Our system is based on a pipeline which first generates a sequence of basic database operations, called a sketch, from a natural language question, then specializes the sketch into a concrete query program with mentions of entities, attributes and relations, and finally executes the program against the database.This pipeline decomposition approach enables us to train the system by leveraging out-of-domain data and semi-synthetic data generated by GPT-3, thus reducing overfitting and shortcut learning even with limited amount of in-domain training data.
Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking

There has been significant interest in zero and few-shot learning for dialogue state tracking (DST) due to the high cost of collecting and annotating task-oriented dialogues.Recent work has demonstrated that in-context learning requires very little data and zero parameter updates, and even outperforms trained methods in the few-shot setting (Hu et al., 2022).We propose RefPyDST, which advances the state of the art with three advancements to in-context learning for DST.First, we formulate DST as a Python programming task, explicitly modeling language coreference as variable reference in Python.Second, since in-context learning depends highly on the context examples, we propose a method to retrieve a diverse set of relevant examples to improve performance.Finally, we introduce a novel re-weighting method during decoding that takes into account probabilities of competing surface forms, and produces a more accurate dialogue state prediction.We evaluate our approach using MultiWOZ and achieve state-of-the-art multi-domain joint-goal accuracy in zero and few-shot settings. 1
Contrastive Hierarchical Discourse Graph for Scientific Document Summarization

The extended structural context has made scientific paper summarization a challenging task.This paper proposes CHANGES, a contrastive hierarchical graph neural network for extractive scientific paper summarization.CHANGES represents a scientific paper with a hierarchical discourse graph and learns effective sentence representations with dedicated designed hierarchical graph information aggregation.We also propose a graph contrastive learning module to learn global theme-aware sentence representations.Extensive experiments on the PubMed and arXiv benchmark datasets prove the effectiveness of CHANGES and the importance of capturing hierarchical structure information in modeling scientific papers.
Typo-Robust Representation Learning for Dense Retrieval

Dense retrieval is a basic building block of information retrieval applications.One of the main challenges of dense retrieval in real-world settings is the handling of queries containing misspelled words.A popular approach for handling misspelled queries is minimizing the representations discrepancy between misspelled queries and their pristine ones.Unlike the existing approaches, which only focus on the alignment between misspelled and pristine queries, our method also improves the contrast between each misspelled query and its surrounding queries.To assess the effectiveness of our proposed method, we compare it against the existing competitors using two benchmark datasets and two base encoders.Our method outperforms the competitors in all cases with misspelled queries.Our code and models are available at https://github.com/panuthept/DST-DenseRetrieval.
VendorLink: An NLP approach for Identifying &amp; Linking Vendor Migrants &amp; Potential Aliases on Darknet Markets

The anonymity on the Darknet allows vendors to stay undetected by using multiple vendor aliases or frequently migrating between markets.Consequently, illegal markets and their connections are challenging to uncover on the Darknet.To identify relationships between illegal markets and their vendors, we propose Ven-dorLink, an NLP-based approach that examines writing patterns to verify, identify, and link unique vendor accounts across text advertisements (ads) on seven public Darknet markets.In contrast to existing literature, VendorLink utilizes the strength of supervised pre-training to perform closed-set vendor verification, openset vendor identification, and low-resource market adaption tasks.Through VendorLink, we uncover (i) 15 migrants and 71 potential aliases in the Alphabay-Dreams-Silk dataset, (ii) 17 migrants and 3 potential aliases in the Valhalla-Berlusconi dataset, and (iii) 75 migrants and 10 potential aliases in the Traderoute-Agora dataset.Altogether, our approach can help Law Enforcement Agencies (LEA) make more informed decisions by verifying and identifying migrating vendors and their potential aliases on existing and Low-Resource (LR) emerging Darknet markets.
Opinion Tree Parsing for Aspect-based Sentiment Analysis

Extracting sentiment elements using pretrained generative models has recently led to large improvements in aspect-based sentiment analysis benchmarks.However, these models always need large-scale computing resources, and they also ignore explicit modeling of structure between sentiment elements.To address these challenges, we propose an opinion tree parsing model, aiming to parse all the sentiment elements from an opinion tree, which is much faster, and can explicitly reveal a more comprehensive and complete aspect-level sentiment structure.In particular, we first introduce a novel context-free opinion grammar to normalize the opinion tree structure.We then employ a neural chart-based opinion tree parser to fully explore the correlations among sentiment elements and parse them into an opinion tree structure.Extensive experiments show the superiority of our proposed model and the capacity of the opinion tree parser with the proposed context-free opinion grammar.More importantly, the results also prove that our model is much faster than previous models.Our code can be found in https://github.com/HoraceXIaoyiBao/OTP4ABSA-ACL2023.
DIP: Dead code Insertion based Black-box Attack for Programming Language Model

Automatic processing of source code, such as code clone detection and software vulnerability detection, is very helpful to software engineers.Large pre-trained Programming Language (PL) models (such as CodeBERT, Graph-CodeBERT, CodeT5, etc.), show very powerful performance on these tasks.However, these PL models are vulnerable to adversarial examples that are generated with slight perturbation.Unlike natural language, an adversarial example of code must be semantic-preserving and compilable.Due to the requirements, it is hard to directly apply the existing attack methods for natural language models.In this paper, we propose DIP (Dead code Insertion based Blackbox Attack for Programming Language Model), a high-performance and efficient black-box attack method to generate adversarial examples using dead code insertion.We evaluate our proposed method on 9 victim downstream-task large code models.Our method outperforms the state-of-the-art black-box attack in both attack efficiency and attack quality, while generated adversarial examples are compiled preserving semantic functionality.
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model

Pretrained language models have achieved remarkable success in various natural language processing tasks.However, pretraining has recently shifted toward larger models and larger data, and this has resulted in significant computational and energy costs.In this paper, we propose Influence Subset Selection (ISS) for language model, which explicitly utilizes end-task knowledge to select a tiny subset of the pretraining corpus.Specifically, the ISS selects the samples that will provide the most positive influence on the performance of the end-task.Furthermore, we design a gradient matching based influence estimation method, which can drastically reduce the computation time of influence.With only 0.45% of the data and a three-orders-of-magnitude lower computational cost, ISS outperformed pretrained models (e.g., RoBERTa) on eight datasets covering four domains.
History Semantic Graph Enhanced Conversational KBQA with Temporal Information Modeling

Context information modeling is an important task in conversational KBQA.However, existing methods usually assume the independence of utterances and model them in isolation.In this paper, we propose a History Semantic Graph Enhanced KBQA model (HSGE) that is able to effectively model long-range semantic dependencies in conversation history while maintaining low computational cost.The framework incorporates a context-aware encoder, which employs a dynamic memory decay mechanism and models context at different levels of granularity.We evaluate HSGE on a widely used benchmark dataset for complex sequential question answering.Experimental results demonstrate that it outperforms existing baselines averaged on all question types.
AMR-based Network for Aspect-based Sentiment Analysis

Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment classification task.Many recent works have used dependency trees to extract the relation between aspects and contexts and have achieved significant improvements.However, further improvement is limited due to the potential mismatch between the dependency tree as a syntactic structure and the sentiment classification as a semantic task.To alleviate this gap, we replace the syntactic dependency tree with the semantic structure named Abstract Meaning Representation (AMR) and propose a model called AMR-based Path Aggregation Relational Network (APARN) to take full advantage of semantic structures.In particular, we design the path aggregator and the relation-enhanced selfattention mechanism that complement each other.The path aggregator extracts semantic features from AMRs under the guidance of sentence information, while the relationenhanced self-attention mechanism in turn improves sentence features with refined semantic information.Experimental results on four public datasets demonstrate 1.13% average F1 improvement of APARN in ABSA when compared with state-of-the-art baselines. 1
Enhancing Event Causality Identification with Counterfactual Reasoning

Existing methods for event causality identification (ECI) focus on mining potential causal signals, i.e., causal context keywords and event pairs.However, causal signals are ambiguous, which may lead to the context-keywords bias and the event-pairs bias.To solve this issue, we propose the counterfactual reasoning that explicitly estimates the influence of context keywords and event pairs in training, so that we are able to eliminate the biases in inference.Experiments are conducted on two datasets, the result demonstrates the effectiveness of our method.
Evaluating Embedding APIs for Information Retrieval

The ever-increasing size of language models curtails their widespread availability to the community, thereby galvanizing many companies into offering access to large language models through APIs.One particular type, suitable for dense retrieval, is a semantic embedding service that builds vector representations of input text.With a growing number of publicly available APIs, our goal in this paper is to analyze existing offerings in realistic retrieval scenarios, to assist practitioners and researchers in finding suitable services according to their needs.Specifically, we investigate the capabilities of existing semantic embedding APIs on domain generalization and multilingual retrieval.For this purpose, we evaluate these services on two standard benchmarks, BEIR and MIRACL.We find that re-ranking BM25 results using the APIs is a budget-friendly approach and is most effective in English, in contrast to the standard practice of employing them as first-stage retrievers.For non-English retrieval, re-ranking still improves the results, but a hybrid model with BM25 works best, albeit at a higher cost.We hope our work lays the groundwork for evaluating semantic embedding APIs that are critical in search and more broadly, for information access.
Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming

Pre-trained large language models (LMs) struggle to perform logical reasoning reliably despite advances in scale and compositionality.In this work, we tackle this challenge through the lens of symbolic programming.We propose DSR-LM, a Differentiable Symbolic Reasoning framework where pre-trained LMs govern the perception of factual knowledge, and a symbolic module performs deductive reasoning.In contrast to works that rely on hand-crafted logic rules, our differentiable symbolic reasoning framework efficiently learns weighted rules and applies semantic loss to further improve LMs.DSR-LM is scalable, interpretable, and allows easy integration of prior knowledge, thereby supporting extensive symbolic programming to robustly derive a logical conclusion.The results of our experiments suggest that DSR-LM improves the logical reasoning abilities of pre-trained language models, resulting in a significant increase in accuracy of over 20% on deductive reasoning benchmarks.Furthermore, DSR-LM outperforms a variety of competitive baselines when faced with systematic changes in sequence length. 1
JCT at SemEval-2023 Tasks 12 A and 12B: Sentiment Analysis for Tweets Written in Low-resource African Languages using Various Machine Learning and Deep Learning Methods, Resampling, and HyperParameter Tuning

In this paper, we describe our submissions to the SemEval-2023 contest.We tackled subtask 12 -"AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset".We developed different models for 12 African languages and a 13 th model for a multilingual dataset built from these 12 languages.We applied a wide variety of word and char n-grams based on their tfidf values, 4 classical machine learning methods, 2 deep learning methods, and 3 oversampling methods.We used 12 sentiment lexicons and applied extensive hyperparameter tuning.
Team Cadence at MEDIQA-Chat 2023: Generating, augmenting and summarizing clinical dialogue with large language models

This paper describes Team Cadence's winning submission to Task C of the MEDIQA-Chat 2023 shared tasks.We also present the set of methods, including a novel N-pass strategy to summarize a mix of clinical dialogue and an incomplete summarized note, used to complete Task A and Task B, ranking highly on the leaderboard amongst stable and reproducible code submissions.The shared tasks invited participants to summarize, classify and generate patient-doctor conversations.Considering the small volume of training data available, we took a data-augmentation-first approach to the three tasks by focusing on the dialogue generation task, i.e., Task C. It proved effective in improving our models' performance on Task A and Task B. We also found the BART architecture to be highly versatile, as it formed the base for all our submissions.Finally, based on the results shared by the organizers, we note that Team Cadence was the only team to submit stable and reproducible runs to all three tasks.
MDC at BioLaySumm Task 1: Evaluating GPT Models for Biomedical Lay Summarization

This paper presents our approach to the Bi-oLaySumm Task 1 shared task, held at the BioNLP 2023 Workshop.The effective communication of scientific knowledge to the general public is often limited by the technical language used in research, making it difficult for non-experts to comprehend.To address this issue, lay summaries can be used to explain research findings to non-experts in an accessible form.We conduct an evaluation of autoregressive language models, both general and specialized for the biomedical domain, to generate lay summaries from biomedical research article abstracts.Our findings demonstrate that a GPT-3.5 model combined with a straightforward few-shot prompt produces lay summaries that achieve significantly higher relevance and factuality compared to those generated by a fine-tuned BioGPT model.However, the summaries generated by the BioGPT model exhibit better readability.Notably, our submission for the shared task achieved 1st place in the competition.
Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages

While impressive performance has been achieved on the task of Answer Sentence Selection (AS2) for English, the same does not hold for languages that lack large labeled datasets.In this work, we propose Cross-Lingual Knowledge Distillation (CLKD) from a strong English AS2 teacher as a method to train AS2 models for low-resource languages in the tasks without the need of labeled data for the target language.To evaluate our method, we introduce 1) Xtr-WikiQA, 1 a translation-based Wik-iQA dataset for 9 additional languages, and 2) TyDi-AS2, 2 a multilingual AS2 dataset with over 70K questions spanning 8 typologically diverse languages.We conduct extensive experiments on Xtr-WikiQA and TyDi-AS2 with multiple teachers, diverse monolingual and multilingual pretrained language models (PLMs) as students, and both monolingual and multilingual training.The results demonstrate that CLKD either outperforms or rivals even supervised fine-tuning with the same amount of labeled data and a combination of machine translation and the teacher model.Our method can potentially enable stronger AS2 models for lowresource languages, while TyDi-AS2 can serve as the largest multilingual AS2 dataset for further studies in the research community.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation

Empathy is a crucial factor in open-domain conversations, which naturally shows one's caring and understanding to others.Though several methods have been proposed to generate empathetic responses, existing works often lead to monotonous empathy that refers to generic and safe expressions.In this paper, we propose to use explicit control to guide the empathy expression and design a framework DIFFUSEMP based on conditional diffusion language model to unify the utilization of dialogue context and attribute-oriented control signals.Specifically, communication mechanism, intent, and semantic frame are imported as multi-grained signals that control the empathy realization from coarse to fine levels.We then design a specific masking strategy to reflect the relationship between multi-grained signals and response tokens, and integrate it into the diffusion model to influence the generative process.Experimental results on a benchmark dataset EMPA-THETICDIALOGUE show that our framework outperforms competitive baselines in terms of controllability, informativeness, and diversity without the loss of context-relatedness.
ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis aims to predict the sentiment of video content.Recent research suggests that multimodal sentiment analysis critically depends on learning a good representation of multimodal information, which should contain both modality-invariant representations that are consistent across modalities as well as modality-specific representations.In this paper, we propose ConFEDE, a unified learning framework that jointly performs contrastive representation learning and contrastive feature decomposition to enhance representation of multimodal information.It decomposes each of the three modalities of a video sample, including text, video frames, and audio, into a similarity feature and a dissimilarity feature, which are learned by a contrastive relation centered around text.We conducted extensive experiments on CH-SIMS, MOSI and MOSEI to evaluate various state-of-the-art multimodal sentiment analysis methods.Experimental results show that ConFEDE outperforms all baselines on these datasets on a range of metrics.
Transformer Language Models Handle Word Frequency in Prediction Head

Prediction head is a crucial component of Transformer language models.Despite its direct impact on prediction, this component has often been overlooked in analyzing Transformers.In this study, we investigate the inner workings of the prediction head, specifically focusing on bias parameters.Our experiments with BERT and GPT-2 models reveal that the biases in their word prediction heads play a significant role in the models' ability to reflect word frequency in a corpus, aligning with the logit adjustment method commonly used in long-tailed learning.We also quantify the effect of controlling the biases in practical auto-regressive text generation scenarios; under a particular setting, more diverse text can be generated without compromising text quality.https://github.com/gorokoba560/transformer-lm-word-freq-bias
BREAK: Breaking the Dialogue State Tracking Barrier with Beam Search and Re-ranking

Despite the recent advances in dialogue state tracking (DST), the joint goal accuracy (JGA) of the existing methods on MultiWOZ 2.1 still remains merely 60%.In our preliminary error analysis, we find that beam search produces a pool of candidates that is likely to include the correct dialogue state.Motivated by this observation, we introduce a novel framework, called BREAK (Beam search and RE-rAnKing), that achieves outstanding performance on DST.Our proposed method performs DST in two stages: (i) generating k-best dialogue state candidates with beam search and (ii) re-ranking the candidates to select the correct dialogue state.This simple yet powerful framework shows state-of-the-art performance on all versions of MultiWOZ and M2M datasets.Most notably, we push the joint goal accuracy to 80-90% on MultiWOZ 2.1-2.4,which is an improvement of 23.6%, 26.3%, 21.7%, and 10.8% over the previous best-performing models, respectively.The data and code will be available at
Do Large Language Models Know What They Don’t Know?

Large language models (LLMs) have a wealth of knowledge that allows them to excel in various Natural Language Processing (NLP) tasks.Current research focuses on enhancing their performance within their existing knowledge.Despite their vast knowledge, LLMs are still limited by the amount of information they can accommodate and comprehend.Therefore, the ability to understand their own limitations on the unknows, referred to as self-knowledge, is of paramount importance.This study aims to evaluate LLMs' self-knowledge by assessing their ability to identify unanswerable or unknowable questions.We introduce an automated methodology to detect uncertainty in the responses of these models, providing a novel measure of their self-knowledge.We further introduce a unique dataset, SelfAware, consisting of unanswerable questions from five diverse categories and their answerable counterparts.Our extensive analysis, involving 20 LLMs including GPT-3, InstructGPT, and LLaMA, discovering an intrinsic capacity for self-knowledge within these models.Moreover, we demonstrate that in-context learning and instruction tuning can further enhance this self-knowledge.Despite this promising insight, our findings also highlight a considerable gap between the capabilities of these models and human proficiency in recognizing the limits of their knowledge.