Reviewerly/exHarmony huggingface dataset:
Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment

Recently, speech-text pre-training methods have shown remarkable success in many speech and natural language processing tasks.However, most previous pre-trained models are usually tailored for one or two specific tasks, but fail to conquer a wide range of speech-text tasks.In addition, existing speech-text pretraining methods fail to explore the contextual information within a dialogue to enrich utterance representations.In this paper, we propose Speech-text dialog Pre-training for spoken dialog understanding with ExpliCiT cRoss-Modal Alignment (SPECTRA), which is the first-ever speech-text dialog pre-training model.Concretely, to consider the temporality of speech modality, we design a novel temporal position prediction task to capture the speech-text alignment.This pre-training task aims to predict the start and end time of each textual word in the corresponding speech waveform.In addition, to learn the characteristics of spoken dialogs, we generalize a response selection task from textual dialog pre-training to speech-text dialog pre-training scenarios.Experimental results on four different downstream speech-text tasks demonstrate the superiority of SPECTRA in learning speech-text alignment and multi-turn dialog context.
A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Conditional inference on joint textual and visual clues is a multi-modal reasoning task that textual clues provide prior permutation or external knowledge, which are complementary with visual content and pivotal to deducing the correct option.Previous methods utilizing pretrained vision-language models (VLMs) have achieved impressive performances, yet they show a lack of multimodal context reasoning capability, especially for text-modal information.To address this issue, we propose a Multi-modal Context Reasoning approach, named ModCR.Compared to VLMs performing reasoning via cross modal semantic alignment, it regards the given textual abstract semantic and objective image information as the pre-context information and embeds them into the language model to perform context reasoning.Different from recent vision-aided language models used in natural language processing, ModCR incorporates the multi-view semantic alignment information between language and vision by introducing the learnable alignment prefix between image and text in the pretrained language model.This makes the language model well-suitable for such multi-modal reasoning scenario on joint textual and visual clues.We conduct extensive experiments on two corresponding data sets and experimental results show significantly improved performance (exact gain by 4.8% on PMR test set) compared to previous strong baselines.
Enhancing Multilingual Document-Grounded Dialogue Using Cascaded Prompt-Based Post-Training Models

The Dialdoc23 shared task presents a Multilingual Document-Grounded Dialogue Systems (MDGDS) challenge, where system responses are generated in multiple languages using user's queries, historical dialogue records and relevant passages.A major challenge for this task is the limited training data available in low-resource languages such as French and Vietnamese.In this paper, we propose Cascaded Promptbased Post-training Models, dividing the task into three subtasks: Retrieval, Reranking and Generation.We conduct post-training on highresource language such as English and Chinese to enhance performance of low-resource languages by using the similarities of languages.Additionally, we utilize the prompt method to activate model's ability on diverse languages within the dialogue domain and explore which prompt is a good prompt.Our comprehensive experiments demonstrate the effectiveness of our proposed methods, which achieved the first place on the leaderboard with a total score of 215.40 in token-level F1, SacreBleu, and Rouge-L metrics.
Multi-granularity Temporal Question Answering over Knowledge Graphs

Recently, question answering over temporal knowledge graphs (i.e., TKGQA) has been introduced and investigated, in quest of reasoning about dynamic factual knowledge.To foster research on TKGQA, a few datasets have been curated (e.g., CRONQUESTIONS and Complex-CRONQUESTIONS), and various models have been proposed based on these datasets.Nevertheless, existing efforts overlook the fact that real-life applications of TKGQA also tend to be complex in temporal granularity, i.e., the questions may concern mixed temporal granularities (e.g., both day and month).To overcome the limitation, in this paper, we motivate the notion of multi-granularity temporal question answering over knowledge graphs and present a largescale dataset for multi-granularity TKGQA, namely MULTITQ.To the best of our knowledge, MULTITQ is among the first of its kind, and compared with existing datasets on TKGQA, MULTITQ features at least two desirable aspects-ample relevant facts and multiple temporal granularities.It is expected to better reflect real-world challenges, and serve as a test bed for TKGQA models.In addition, we propose a competing baseline MultiQA over MULTITQ, which is experimentally demonstrated to be effective in dealing with TKGQA.The data and code are released at https: //github.com/czy1999/MultiTQ.
Pre-trained Language Models Can be Fully Zero-Shot Learners

How can we extend a pre-trained model to many language understanding tasks, without labeled or additional unlabeled data? Pre-trained language models (PLMs) have been effective for a wide range of NLP tasks. However, existing approaches either require fine-tuning on downstream labeled datasets or manually constructing proper prompts. In this paper, we propose nonparametric prompting PLM (NPPrompt) for fully zero-shot language understanding. Unlike previous methods, NPPrompt uses only pre-trained language models and does not require any labeled data or additional raw corpus for further fine-tuning, nor does it rely on humans to construct a comprehensive set of prompt label words. We evaluate NPPrompt against previous major few-shot and zero-shot learning methods on diverse NLP tasks: including text classification, text entailment, similar text retrieval, paraphrasing, and multiple-choice question answering. Experimental results demonstrate that our NPPrompt outperforms the previous best fully zero-shot method by big margins, with absolute gains of 12.8% in accuracy on text classification and 15.6% on the GLUE benchmark. Our source code is available at https://anonymous.4open. science/r/NPPrompt.
Multiview Identifiers Enhanced Generative Retrieval

Instead of simply matching a query to preexisting passages, generative retrieval generates identifier strings of passages as the retrieval target.At a cost, the identifier must be distinctive enough to represent a passage.Current approaches use either a numeric ID or a text piece (such as a title or substrings) as the identifier.However, these identifiers cannot cover a passage's content well.As such, we are motivated to propose a new type of identifier, synthetic identifiers, that are generated based on the content of a passage and could integrate contextualized information that text pieces lack.Furthermore, we simultaneously consider multiview identifiers, including synthetic identifiers, titles, and substrings.These views of identifiers complement each other and facilitate the holistic ranking of passages from multiple perspectives.We conduct a series of experiments on three public datasets, and the results indicate that our proposed approach performs the best in generative retrieval, demonstrating its effectiveness and robustness.The code is released at https://github.com/liyongqi67/MINDER.
CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models

Warning: This paper contains content that may be offensive or upsetting.Pretrained conversational agents have been exposed to safety issues, exhibiting a range of stereotypical human biases such as gender bias.However, there are still limited bias categories in current research, and most of them only focus on English.In this paper, we introduce a new Chinese dataset, CHBias, for bias evaluation and mitigation of Chinese conversational language models.Apart from those previous well-explored bias categories, CHBias includes under-explored bias categories, such as ageism and appearance biases, which received less attention.We evaluate two popular pretrained Chinese conversational models, CDial-GPT and EVA2.0, using CHBias.Furthermore, to mitigate different biases, we apply several debiasing methods to the Chinese pretrained models.Experimental results show that these Chinese pretrained models are potentially risky for generating texts that contain social biases, and debiasing methods using the proposed dataset can make response generation less biased while preserving the models' conversational capabilities.
End-to-end Aspect-based Sentiment Analysis with Combinatory Categorial Grammar

End-to-end Aspect-based Sentiment Analysis (EASA) is a natural language processing (NLP) task that involves extracting aspect terms and identifying the sentiments for them, which provides a fine-grained level of text analysis and thus requires a deep understanding of the running text.Many previous studies leverage advanced text encoders to extract context information and use syntactic information, e.g., the dependency structure of the input sentence, to improve the model performance.However, such models may reach a bottleneck since the dependency structure is not designed to provide semantic information of the text, which is also important for identifying the sentiment and thus leave room for further improvement.Considering that combinatory categorial grammar (CCG) is a formalism that expresses both syntactic and semantic information of a sentence, it has the potential to be beneficial to EASA.In this paper, we propose a novel approach to improve EASA with CCG supertags, which carry the syntactic and semantic information of the associated words and serve as the most important part of the CCG derivation.Specifically, our approach proposes a CCG supertag decoding process to learn the syntactic and semantic information carried by CCG supertags and use the information to guide the attention over the input words so as to identify important contextual information for EASA.Furthermore, a gate mechanism is used in incorporating the weighted contextual information into the backbone EASA decoding process.We evaluate our approach on three publicly available English datasets for EASA, and show that it outperforms strong baselines and achieves state-of-the-art results on all datasets. 1
Joint Document-Level Event Extraction via Token-Token Bidirectional Event Completed Graph

We solve the challenging document-level event extraction problem by proposing a joint exaction methodology that can avoid inefficiency and error propagation issues in classic pipeline methods.Essentially, we address the three crucial limitations in existing studies.First, the autoregressive strategy of path expansion heavily relies on the orders of argument roles.Second, the number of events in documents must be specified in advance.Last, unexpected errors usually exist when decoding events based on the entity-entity adjacency matrix.This paper designs a Token-Token Bidirectional Event Completed Graph (TT-BECG) in which the relation eType-Role 1 -Role 2 serves as the edge type, precisely revealing which tokens play argument roles in an event of a specific event type.Exploiting the token-token adjacency matrix of the TT-BECG, we develop an edge-enhanced joint document-level event extraction model.Guided by the target token-token adjacency matrix, the predicted token-token adjacency matrix can be obtained during model training.Then, the event records in a document are decoded based on the predicted matrix, including the graph structure and edge-type decoding.Extensive experiments are conducted on two public datasets, and the results confirm the effectiveness of our method and its superiority over the state-of-the-art baselines.
A Survey for Efficient Open Domain Question Answering

Open domain question answering (ODQA) is a longstanding task aimed at answering factual questions from a large knowledge corpus without any explicit evidence in natural language processing (NLP).Recent works have predominantly focused on improving the answering accuracy and have achieved promising progress.However, higher accuracy often requires more memory consumption and inference latency, which might not necessarily be efficient enough for direct deployment in the real world.Thus, a trade-off between accuracy, memory consumption and processing speed is pursued.In this paper, we will survey recent advancements in the efficiency of ODQA models and conclude core techniques for achieving efficiency.Additionally, we will provide a quantitative analysis of memory cost, query speed, accuracy, and overall performance comparison.Our goal is to keep scholars informed of the latest advancements and open challenges in ODQA efficiency research and contribute to the further development of ODQA efficiency.
Learning by Analogy: Diverse Questions Generation in Math Word Problem

Solving math word problem (MWP) with AI techniques has recently made great progress with the success of deep neural networks (DNN), but it is far from being solved.We argue that the ability of learning by analogy is essential for an MWP solver to better understand same problems which may typically be formulated in diverse ways.However most existing works exploit the shortcut learning to train MWP solvers simply based on samples with a single question.In lack of diverse questions, these methods merely learn shallow heuristics.In this paper, we make a first attempt to solve MWPs by generating diverse yet consistent questions/equations.Given a typical MWP including the scenario description, question, and equation (i.e., answer), we first generate multiple consistent equations via a group of heuristic rules.We then feed them to a question generator together with the scenario to obtain the corresponding diverse questions, forming a new MWP with a variety of questions and equations.Finally we engage a data filter to remove those unreasonable MWPs, keeping the high-quality augmented ones.To evaluate the ability of learning by analogy for an MWP solver, we generate a new MWP dataset (called DiverseMath23K) with diverse questions by extending the current benchmark Math23K.Extensive experimental results demonstrate that our proposed method can generate high-quality diverse questions with corresponding equations, further leading to performance improvement on Diverse-Math23K.The code and dataset is available at: https://github.com/zhouzihao501/DiverseMWP.
Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering

The integration of multi-document pre-training objectives into language models has resulted in remarkable improvements in multi-document downstream tasks.In this work, we propose extending this idea by pre-training a generic multi-document model from a novel crossdocument question answering pre-training objective.To that end, given a set (or cluster) of topically-related documents, we systematically generate semantically-oriented questions from a salient sentence in one document and challenge the model, during pre-training, to answer these questions while "peeking" into other topically-related documents.In a similar manner, the model is also challenged to recover the sentence from which the question was generated, again while leveraging cross-document information.This novel multidocument QA formulation directs the model to better recover cross-text informational relations, and introduces a natural augmentation that artificially increases the pre-training data.Further, unlike prior multi-document models that focus on either classification or summarization tasks, our pre-training objective formulation enables the model to perform tasks that involve both short text generation (e.g., QA) and long text generation (e.g., summarization).Following this scheme, we pre-train our model -termed QAMDEN -and evaluate its performance across several multi-document tasks, including multi-document QA, summarization, and query-focused summarization, yielding improvements of up to 7%, and significantly outperforms zero-shot GPT-3.5 and GPT-4. 1
Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation

Image-to-text tasks, such as open-ended image captioning and controllable image description, have received extensive attention for decades.Here, we further advance this line of work by presenting Visual Spatial Description (VSD), a new perspective for image-to-text toward spatial semantics.Given an image and two objects inside it, VSD aims to produce one description focusing on the spatial perspective between the two objects.Accordingly, we manually annotate a dataset to facilitate the investigation of the newly-introduced task and build several benchmark encoder-decoder models by using VL-BART and VL-T5 as backbones.In addition, we investigate pipeline and joint end-to-end architectures for incorporating visual spatial relationship classification (VSRC) information into our model.Finally, we conduct experiments on our benchmark dataset to evaluate all our models.Results show that our models are impressive, providing accurate and human-like spatial-oriented text descriptions.Meanwhile, VSRC has great potential for VSD, and the joint end-to-end architecture is the better choice for their integration.We make the dataset and codes public for research purposes.
UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression

Geometry problem solving is a well-recognized testbed for evaluating the high-level multimodal reasoning capability of deep models.In most existing works, two main geometry problems: calculation and proving, are usually treated as two specific tasks, hindering a deep model to unify its reasoning capability on multiple math tasks.However, in essence, these two tasks have similar problem representations and overlapped math knowledge which can improve the understanding and reasoning ability of a deep model on both two tasks.Therefore, we construct a large-scale Unified Geometry problem benchmark, UniGeo, which contains 4,998 calculation problems and 9,543 proving problems.Each proving problem is annotated with a multi-step proof with reasons and mathematical expressions.The proof can be easily reformulated as a proving sequence that shares the same formats with the annotated program sequence for calculation problems.Naturally, we also present a unified multitask Geometric Transformer framework, Geoformer, to tackle calculation and proving problems simultaneously in the form of sequence generation, which finally shows the reasoning ability can be improved on both two tasks by unifying formulation.Furthermore, we propose a Mathematical Expression Pretraining (MEP) method that aims to predict the mathematical expressions in the problem solution, thus improving the Geoformer model.Experiments on the UniGeo demonstrate that our proposed Geoformer obtains state-of-the-art performance by outperforming task-specific model NGS with over 5.6% and 3.2% accuracies on calculation and proving problems, respectively.
SetGNER: General Named Entity Recognition as Entity Set Generation

Recently, joint recognition of flat, nested and discontinuous entities has received increasing attention.Motivated by the observation that the target output of NER is essentially a set of sequences, we propose a novel entity set generation framework for general NER scenes in this paper.Different from sequence-to-sequence NER methods, our method does not force the entities to be generated in a predefined order and can get rid of the problem of error propagation and inefficient decoding.Distinguished from the set-prediction NER framework, our method treats each entity as a sequence and is capable of recognizing discontinuous mentions.Given an input sentence, the model first encodes the sentence in word-level and detects potential entity mentions based on the encoder's output, then reconstructs entity mentions from the detected entity heads in parallel.To let the encoder of our model capture better rightto-left semantic structure, we also propose an auxiliary Inverse Generation Training task.Extensive experiments show that our model (w/o.Inverse Generation Training) outperforms stateof-the-art generative NER models by a large margin on two discontinuous NER datasets, two nested NER datasets and one flat NER dataset.Besides, the auxiliary Inverse Generation Training task is found to further improve the model's performance on the five datasets.
Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference

While large pre-trained language models are powerful, their predictions often lack logical consistency across test inputs.For example, a state-of-the-art Macaw question-answering (QA) model answers Yes to Is a sparrow a bird? and Does a bird have feet?but answers No to Does a sparrow have feet?.To address this failure mode, we propose a framework, Consistency Correction through Relation Detection, or ConCoRD, for boosting the consistency and accuracy of pre-trained NLP models using pre-trained natural language inference (NLI) models without fine-tuning or re-training.Given a batch of test inputs, ConCoRD samples several candidate outputs for each input and instantiates a factor graph that accounts for both the model's belief about the likelihood of each answer choice in isolation and the NLI model's beliefs about pair-wise answer choice compatibility.We show that a weighted MaxSAT solver can efficiently compute highquality answer choices under this factor graph, improving over the raw model's predictions.Our experiments demonstrate that ConCoRD consistently boosts accuracy and consistency of off-the-shelf closed-book QA and VQA models using off-the-shelf NLI models, notably increasing accuracy of LXMERT on ConVQA by 5% absolute.See the project website 1 for code and data.
Training Language Models with Memory Augmentation

Recent work has improved language models (LMs) remarkably by equipping them with a non-parametric memory component.However, most existing approaches only introduce memories at testing time or represent them using a separately trained encoder, resulting in suboptimal training of the language model.In this work, we present TRIME, a novel yet simple training approach designed for training LMs with memory augmentation.Our approach uses a training objective that directly takes inbatch examples as accessible memory.We also present new methods for memory construction and data batching, which are used for adapting to different sets of memories-local, longterm, and external memory-at testing time.We evaluate TRIME on multiple language modeling and machine translation benchmarks and show that it is able to achieve significant improvements across all the settings.Concretely, TRIME reduces the perplexity from 18.70 to 15.37 on WIKITEXT-103, by effectively leveraging a large memory set from the training corpus.Compared to standard LM training, TRIME adds negligible computational overhead and is compatible with different neural architectures, making it a versatile solution for training memory-augmented LMs.