Reviewerly/exHarmony huggingface dataset:
FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue

Pre-trained language models based on general text enable huge success in the NLP scenario.But the intrinsical difference of linguistic patterns between general text and task-oriented dialogues makes existing pre-trained language models less useful in practice.Current dialogue pre-training methods rely on a contrastive framework and face the challenges of both selecting true positives and hard negatives.In this paper, we propose a novel dialogue pretraining model, FutureTOD, which distills future knowledge to the representation of the previous dialogue context using a self-training framework.Our intuition is that a good dialogue representation both learns local context information and predicts future information.Extensive experiments on diverse downstream dialogue tasks demonstrate the effectiveness of our model, especially the generalization, robustness, and learning discriminative dialogue representations capabilities.
Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?

Prompt tuning (PT) which only tunes the embeddings of an additional sequence of tokens per task, keeping the pre-trained language model (PLM) frozen, has shown remarkable performance in few-shot learning.Despite this, PT has been shown to rely heavily on good initialization of the prompt embeddings.In this work, we study meta prompt tuning (MPT) to systematically explore how meta-learning can help improve (if it can) cross-task generalization in PT through learning to initialize the prompt embeddings from other relevant tasks.We empirically analyze a representative set of meta learning algorithms in a wide range of adaptation settings with different source/target task configurations on a large set of few-shot tasks.With extensive experiments and analysis, we demonstrate the effectiveness of MPT.We find the improvement to be significant particularly on classification tasks.For other kinds of tasks such as question answering, we observe that while MPT can outperform PT in most cases, it does not always outperform multi-task learning.We further provide an in-depth analysis from the perspective of task similarity.
KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding

With the advent of pretrained language models (LMs), increasing research efforts have been focusing on infusing commonsense and domain-specific knowledge to prepare LMs for downstream tasks.These works attempt to leverage knowledge graphs, the de facto standard of symbolic knowledge representation, along with pretrained LMs.While existing approaches have leveraged external knowledge, it remains an open question how to jointly incorporate knowledge graphs representing varying contexts-from local (e.g., sentence), to document-level, to global knowledge-to enable knowledge-rich exchange across these contexts.Such rich contextualization can be especially beneficial for long document understanding tasks since standard pretrained LMs are typically bounded by the input sequence length.In light of these challenges, we propose KALM, a Knowledge-Aware Language Model that jointly leverages knowledge in local, document-level, and global contexts for long document understanding.KALM first encodes long documents and knowledge graphs into the three knowledge-aware context representations.It then processes each context with context-specific layers, followed by a "context fusion" layer that facilitates knowledge exchange to derive an overarching document representation.Extensive experiments demonstrate that KALM achieves state-of-the-art performance on six long document understanding tasks and datasets.Further analyses reveal that the three knowledge-aware contexts are complementary and they all contribute to model performance, while the importance and information exchange patterns of different contexts vary with respect to different tasks and datasets.
MUX-PLMs: Pre-training Language Models with Data Multiplexing

The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies.The burgeoning cost of inference for everincreasing model sizes coupled with hardware shortages has limited affordable access and poses a pressing need for efficiency approaches geared towards high throughput and performance.Multi-input multi-output (MIMO) algorithms such as data multiplexing, offer a promising solution with a many-fold increase in throughput by performing inference for multiple inputs at the cost of a single input.Yet these approaches are not currently performant enough to be deployed in modern systems.We change that by developing MUX-PLMs, a class of high throughput pre-trained language models (PLMs) trained with data multiplexing, that can be fine-tuned for any downstream task to yield high-throughput high-performance.Our novel multiplexing and demultiplexing modules proficiently entangle and disentangle inputs, and enable high-performance high throughput MUX-PLMs that are competitive with vanilla PLMs while achieving 2x/5x inference speedup with only a 1 -4% drop on a broad suite of tasks.
Attributable and Scalable Opinion Summarization

We propose a method for unsupervised opinion summarization that encodes sentences from customer reviews into a hierarchical discrete latent space, then identifies common opinions based on the frequency of their encodings.We are able to generate both abstractive summaries by decoding these frequent encodings, and extractive summaries by selecting the sentences assigned to the same frequent encodings.Our method is attributable, because the model identifies sentences used to generate the summary as part of the summarization process.It scales easily to many hundreds of input reviews, because aggregation is performed in the latent space rather than over long sequences of tokens.We also demonstrate that our appraoch enables a degree of control, generating aspectspecific summaries by restricting the model to parts of the encoding space that correspond to desired aspects (e.g., location or food).Automatic and human evaluation on two datasets from different domains demonstrates that our method generates summaries that are more informative than prior work and better grounded in the input reviews.
Residual Prompt Tuning: improving prompt tuning with residual reparameterization

Prompt tuning is one of the successful approaches for parameter-efficient tuning of pretrained language models.Despite being arguably the most parameter-efficient (tuned soft prompts constitute < 0.1% of total parameters), it typically performs worse than other efficient tuning methods and is quite sensitive to hyper-parameters.In this work, we introduce RESIDUAL PROMPT TUNING -a simple and efficient method that significantly improves the performance and stability of prompt tuning.We propose to reparameterize soft prompt embeddings using a shallow network with a residual connection.Our experiments show that RESIDUAL PROMPT TUNING significantly outperforms prompt tuning on SuperGLUE benchmark across T5-Large, T5-Base and BERT-Base models.Notably, our method reaches +7 points improvement over prompt tuning with T5-Base and allows to reduce the prompt length by ×10 without hurting performance.In addition, we show that our approach is robust to the choice of learning rate and prompt initialization, and is effective in few-shot settings. 1
Single Sequence Prediction over Reasoning Graphs for Multi-hop QA

Recent generative approaches for multi-hop question answering (QA) utilize the fusion-indecoder method (Izacard and Grave, 2021) to generate a single sequence output which includes both a final answer and a reasoning path taken to arrive at that answer, such as passage titles and key facts from those passages.While such models can lead to better interpretability and high quantitative scores, they often have difficulty accurately identifying the passages corresponding to key entities in the context, resulting in incorrect passage hops and a lack of faithfulness in the reasoning path.To address this, we propose a single-sequence prediction method over a local reasoning graph (SEQGRAPH) 1 that integrates a graph structure connecting key entities in each context passage to relevant subsequent passages for each question.We use a graph neural network to encode this graph structure and fuse the resulting representations into the entity representations of the model.Our experiments show significant improvements in answer exact-match/F1 scores and faithfulness of grounding in the reasoning path on the HotpotQA dataset and achieve stateof-the-art numbers on the Musique dataset with only up to a 4% increase in model parameters.
Resolving Indirect Referring Expressions for Entity Selection

Recent advances in language modeling have enabled new conversational systems.In particular, it is often desirable for people to make choices among specified options when using such systems.We address this problem of reference resolution, when people use natural expressions to choose between the entities.For example, given the choice 'Should we make a Simnel cake or a Pandan cake?' a natural response from a dialog participant may be indirect: 'let's make the green one'.Such natural expressions have been little studied for reference resolution.We argue that robustly understanding such language has large potential for improving naturalness in dialog, recommendation, and search systems.We create AltEntities 1 (Alternative Entities), a new public dataset of 42K entity pairs and expressions (referring to one entity in the pair), and develop models for the disambiguation problem.Consisting of indirect referring expressions across three domains, our corpus enables for the first time the study of how language models can be adapted to this task.We find they achieve 82%-87% accuracy in realistic settings, which while reasonable also invites further advances.
Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization

Speaker diarization(SD) is a classic task in speech processing and is crucial in multi-party scenarios such as meetings and conversations.Current mainstream speaker diarization approaches consider acoustic information only, which result in performance degradation when encountering adverse acoustic conditions.In this paper, we propose methods to extract speaker-related information from semantic content in multi-party meetings, which, as we will show, can further benefit speaker diarization.We introduce two sub-tasks, Dialogue Detection and Speaker-Turn Detection, in which we effectively extract speaker information from conversational semantics.We also propose a simple yet effective algorithm to jointly model acoustic and semantic information and obtain speaker-identified texts.Experiments on both AISHELL-4 and AliMeeting datasets show that our method achieves consistent improvements over acoustic-only speaker diarization systems.
NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models

Prompt-based learning is vulnerable to backdoor attacks.Existing backdoor attacks against prompt-based models consider injecting backdoors into the entire embedding layers or word embedding vectors.Such attacks can be easily affected by retraining on downstream tasks and with different prompting strategies, limiting the transferability of backdoor attacks.In this work, we propose transferable backdoor attacks against prompt-based models, called NOTABLE, which is independent of downstream tasks and prompting strategies.Specifically, NOTABLE injects backdoors into the encoders of PLMs by utilizing an adaptive verbalizer to bind triggers to specific words (i.e., anchors).It activates the backdoor by pasting input with triggers to reach adversary-desired anchors, achieving independence from downstream tasks and prompting strategies.We conduct experiments on six NLP tasks, three popular models, and three prompting strategies.Empirical results show that NOTABLE achieves superior attack performance (i.e., attack success rate over 90% on all the datasets), and outperforms two state-ofthe-art baselines.Evaluations on three defenses show the robustness of NOTABLE.Our code can be found at https://github.com/
Do You Hear The People Sing? Key Point Analysis via Iterative Clustering and Abstractive Summarisation

Argument summarisation is a promising but currently under-explored field.Recent work has aimed to provide textual summaries in the form of concise and salient short texts, i.e., key points (KPs), in a task known as Key Point Analysis (KPA).One of the main challenges in KPA is finding high-quality key point candidates from dozens of arguments even in a small corpus.Furthermore, evaluating key points is crucial in ensuring that the automatically generated summaries are useful.Although automatic methods for evaluating summarisation have considerably advanced over the years, they mainly focus on sentence-level comparison, making it difficult to measure the quality of a summary (a set of KPs) as a whole.Aggravating this problem is the fact that human evaluation is costly and unreproducible.To address the above issues, we propose a two-step abstractive summarisation framework based on neural topic modelling with an iterative clustering procedure, to generate key points which are aligned with how humans identify key points.Our experiments show that our framework advances the state of the art in KPA, with performance improvement of up to 14 (absolute) percentage points, in terms of both ROUGE and our own proposed evaluation metrics 1 .Furthermore, we evaluate the generated summaries using a novel set-based evaluation toolkit.Our quantitative analysis demonstrates the effectiveness of our proposed evaluation metrics in assessing the quality of generated KPs.Human evaluation further demonstrates the advantages of our approach and validates that our proposed evaluation metric is more consistent with human judgment than ROUGE scores. Children express themselves through the clothes they wear and should be able to do this at school School uniform is harming the student's self expressionSchool uniforms are expensive [...] Children should be able to dress as they wish, within reason, at school rather than being restricted from expressing themselves through their clothes.Children should be allowed to express themselves School uniform is unaffordable for many single parents and should be abandoned.School uniforms are an expense that many families can't afford.there are plenty of ways to get very cheap clothing, but discounted uniforms are more difficult to obtain.School uniforms are expensive and puts an undue burden on the parents of the students.School uniforms are expensive for the school and take money from other important programs.School unforms stifle freedom of expression.they can be costly and make circumstances difficult for those on a budget [...
Improved Instruction Ordering in Recipe-Grounded Conversation

In this paper, we study the task of instructional dialogue and focus on the cooking domain.Analyzing the generated output of the GPT-J model, we reveal that the primary challenge for a recipe-grounded dialog system is how to provide the instructions in the correct order.We hypothesize that this is due to the model's lack of understanding of user intent and inability to track the instruction state (i.e., which step was last instructed).Therefore, we propose to explore two auxiliary subtasks, namely User Intent Detection and Instruction State Tracking, to support Response Generation with improved instruction grounding.Experimenting with our newly collected dataset, ChattyChef, shows that incorporating user intent and instruction state information helps the response generation model mitigate the incorrect order issue.Furthermore, to investigate whether Chat-GPT has completely solved this task, we analyze its outputs and find that it also makes mistakes (10.7% of the responses), about half of which are out-of-order instructions.We will release ChattyChef to facilitate further research in this area at: https://github.com/octaviaguo/ChattyChef.
Direct Fact Retrieval from Knowledge Graphs without Entity Linking

There has been a surge of interest in utilizing Knowledge Graphs (KGs) for various natural language processing/understanding tasks.The conventional mechanism to retrieve facts in KGs usually involves three steps: entity span detection, entity disambiguation, and relation classification.However, this approach requires additional labels for training each of the three subcomponents in addition to pairs of input texts and facts, and also may accumulate errors propagated from failures in previous steps.To tackle these limitations, we propose a simple knowledge retrieval framework, which directly retrieves facts from the KGs given the input text based on their representational similarities, which we refer to as Direct Fact Retrieval (Di-FaR).Specifically, we first embed all facts in KGs onto a dense embedding space by using a language model trained by only pairs of input texts and facts, and then provide the nearest facts in response to the input text.Since the fact, consisting of only two entities and one relation, has little context to encode, we propose to further refine ranks of top-k retrieved facts with a reranker that contextualizes the input text and the fact jointly.We validate our Di-FaR framework on multiple fact retrieval tasks, showing that it significantly outperforms relevant baselines that use the three-step approach.
DAMP: Doubly Aligned Multilingual Parser for Task-Oriented Dialogue

Modern virtual assistants use internal semantic parsing engines to convert user utterances to actionable commands.However, prior work has demonstrated multilingual models are less robust for semantic parsing compared to other tasks.In global markets such as India and Latin America, robust multilingual semantic parsing is critical as codeswitching between languages is prevalent for bilingual users.In this work we dramatically improve the zero-shot performance of a multilingual and codeswitched semantic parsing system using two stages of multilingual alignment.First, we show that contrastive alignment pretraining improves both English performance and transfer efficiency.We then introduce a constrained optimization approach for hyperparameter-free adversarial alignment during finetuning.Our Doubly Aligned Multilingual Parser (DAMP) improves mBERT transfer performance by 3x, 6x, and 81x on the Spanglish, Hinglish and Multilingual Task Oriented Parsing benchmarks respectively and outperforms XLM-R and mT5-Large using 3.2x fewer parameters. 1
Rutgers Multimedia Image Processing Lab at SemEval-2023 Task-1: Text-Augmentation-based Approach for Visual Word Sense Disambiguation

This paper describes our system used in SemEval-2023 Task-1: Visual Word Sense Disambiguation (VWSD). The VWSD task is to identify the correct image that corresponds to an ambiguous target word given limited textual context. To reduce word ambiguity and enhance image selection, we proposed several text augmentation techniques, such as prompting, WordNet synonyms, and text generation. We experimented with different vision-language pre-trained models to capture the joint features of the augmented text and image. Our approach achieved the best performance using a combination of GPT-3 text generation and the CLIP model. On the multilingual test sets, our system achieved an average hit rate (at top-1) of 51.11 and a mean reciprocal rank of 65.69.
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants

We present the first unified study of the efficiency of self-attention-based Transformer variants spanning text, speech and vision.We identify input length thresholds (tipping points) at which efficient Transformer variants become more efficient than vanilla models, using a variety of efficiency metrics (latency, throughput, and memory).To conduct this analysis for speech, we introduce L-HuBERT, a novel localattention variant of a self-supervised speech model.We observe that these thresholds are (a) much higher than typical dataset sequence lengths and (b) dependent on the metric and modality, showing that choosing the right model depends on modality, task type (long-form vs. typical context) and resource constraints (time vs. memory).By visualising the breakdown of the computational costs for transformer components, we also show that non-self-attention components exhibit significant computational costs.
MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling

We present MatSci-NLP, a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text.We construct the benchmark from publicly available materials science text data to encompass seven different NLP tasks, including conventional NLP tasks like named entity recognition and relation classification, as well as NLP tasks specific to materials science, such as synthesis action retrieval which relates to creating synthesis procedures for materials.We study various BERT-based models pretrained on different scientific text corpora on MatSci-NLP to understand the impact of pretraining strategies on understanding materials science text.Given the scarcity of high-quality annotated data in the materials science domain, we perform our fine-tuning experiments with limited training data to encourage the generalize across MatSci-NLP tasks.Our experiments in this low-resource training setting show that language models pretrained on scientific text outperform BERT trained on general text.Mat-BERT, a model pretrained specifically on materials science journals, generally performs best for most tasks.Moreover, we propose a unified text-to-schema for multitask learning on MatSci-NLP and compare its performance with traditional fine-tuning methods.In our analysis of different training methods, we find that our proposed text-to-schema methods inspired by question-answering consistently outperform single and multitask NLP fine-tuning methods.The code and datasets are publicly available 1 .
SciReviewGen: A Large-scale Dataset for Automatic Literature Review Generation

Automatic literature review generation is one of the most challenging tasks in natural language processing.Although large language models have tackled literature review generation, the absence of large-scale datasets has been a stumbling block to the progress.We release SciReviewGen, consisting of over 10,000 literature reviews and 690,000 papers cited in the reviews.Based on the dataset, we evaluate recent transformer-based summarization models on the literature review generation task, including Fusion-in-Decoder (Izacard and Grave, 2021) extended for literature review generation.Human evaluation results show that some machine-generated summaries are comparable to human-written reviews, while revealing the challenges of automatic literature review generation such as hallucinations and a lack of detailed information.
A Pilot Study on Dialogue-Level Dependency Parsing for Chinese

Dialogue-level dependency parsing has received insufficient attention, especially for Chinese.To this end, we draw on ideas from syntactic dependency and rhetorical structure theory (RST), developing a high-quality humanannotated corpus, which contains 850 dialogues and 199,803 dependencies.Considering that such tasks suffer from high annotation costs, we investigate zero-shot and fewshot scenarios.Based on an existing syntactic treebank, we adopt a signal-based method to transform seen syntactic dependencies into unseen ones between elementary discourse units (EDUs), where the signals are detected by masked language modeling.Besides, we apply single-view and multi-view data selection to access reliable pseudo-labeled instances.Experimental results show the effectiveness of these baselines.Moreover, we discuss several crucial points about our dataset and approach.