Reviewerly/exHarmony huggingface dataset:
Decoupling Pseudo Label Disambiguation and Representation Learning for Generalized Intent Discovery

Generalized intent discovery aims to extend a closed-set in-domain intent classifier to an open-world intent set including in-domain and out-of-domain intents.The key challenges lie in pseudo label disambiguation and representation learning.Previous methods suffer from a coupling of pseudo label disambiguation and representation learning, that is, the reliability of pseudo labels relies on representation learning, and representation learning is restricted by pseudo labels in turn.In this paper, we propose a decoupled prototype learning framework (DPL) to decouple pseudo label disambiguation and representation learning.Specifically, we firstly introduce prototypical contrastive representation learning (PCL) to get discriminative representations.And then we adopt a prototypebased label disambiguation method (PLD) to obtain pseudo labels.We theoretically prove that PCL and PLD work in a collaborative fashion and facilitate pseudo label disambiguation.Experiments and analysis on three benchmark datasets show the effectiveness of our method. 1
Optimizing Test-Time Query Representations for Dense Retrieval

Recent developments of dense retrieval rely on quality representations of queries and contexts from pre-trained query and context encoders.In this paper, we introduce TOUR (Test-Time Optimization of Query Representations), which further optimizes instance-level query representations guided by signals from testtime retrieval results.We leverage a crossencoder re-ranker to provide fine-grained pseudo labels over retrieval results and iteratively optimize query representations with gradient descent.Our theoretical analysis reveals that TOUR can be viewed as a generalization of the classical Rocchio algorithm for pseudo relevance feedback, and we present two variants that leverage pseudo-labels as hard binary or soft continuous labels.We first apply TOUR on phrase retrieval with our proposed phrase re-ranker, and also evaluate its effectiveness on passage retrieval with an off-the-shelf reranker.TOUR greatly improves end-to-end open-domain question answering accuracy, as well as passage retrieval performance.TOUR also consistently improves direct re-ranking by up to 2.0% while running 1.3-2.4×faster with an efficient implementation.
Constructing Multilingual Code Search Dataset Using Neural Machine Translation

Code search is a task to find programming codes that semantically match the given natural language queries.Even though some of the existing datasets for this task are multilingual on the programming language side, their query data are only in English.In this research, we create a multilingual code search dataset in four natural and four programming languages using a neural machine translation model.Using our dataset, we pre-train and fine-tune the Transformer-based models and then evaluate them on multiple code search test sets.Our results show that the model pre-trained with all natural and programming language data has performed best in most cases.By applying back-translation data filtering to our dataset, we demonstrate that the translation quality affects the model's performance to a certain extent, but the data size matters more.
Learning Query Adaptive Anchor Representation for Inductive Relation Prediction

Relation prediction on knowledge graphs (KGs) attempts to infer the missing links between entities.Most previous studies are limited to the transductive setting where all entities must be seen during the training, making them unable to perform reasoning on emerging entities.Recently, the inductive setting is proposed to handle the entities in the test phase to be unseen during training, However, it suffers from the inefficient reasoning under the enclosing subgraph extraction issue and the lack of effective entity-independent feature modeling.To this end, we propose a novel Query Adaptive Anchor Representation (QAAR) model for inductive relation prediction.First, we extract one opening subgraph and perform reasoning by one time for all candidate triples, which is more efficient when the number of candidate triples is large.Second, we define some query adaptive anchors which are independent on any specific entity.Based on these anchors, we take advantage of the transferable entityindependent features (relation-aware, structureaware and distance features) that can be used to produce entity embeddings for emerging unseen entities.Such entity-independent features is modeled by a query-aware graph attention network on the opening subgraph.Experimental results demonstrate that our proposed QAAR outperforms state-of-the-art baselines in inductive relation prediction task.
WebDP: Understanding Discourse Structures in Semi-Structured Web Documents

Web documents have become rich data resources in current era, and understanding their discourse structure will potentially benefit various downstream document processing applications.Unfortunately, current discourse analysis and document intelligence research mostly focus on either discourse structure of plain text or superficial visual structures in document, which cannot accurately describe discourse structure of highly free-styled and semistructured web documents.To promote discourse studies on web documents, in this paper we introduced a benchmark -WebDP, orienting a new task named Web Document Discourse Parsing.Specifically, a web document discourse structure representation schema is proposed by extending classical discourse theories and adding special features to well represent discourse characteristics of web documents.Then, a manually annotated web document dataset -WEBDOCS is developed to facilitate the study of this parsing task.We compared current neural models on WEBDOCS and experimental results show that WebDP is feasible but also challenging for current models.
Towards Argument-Aware Abstractive Summarization of Long Legal Opinions with Summary Reranking

We propose a simple approach for the abstractive summarization of long legal opinions that considers the argument structure of the document.Legal opinions often contain complex and nuanced argumentation, making it challenging to generate a concise summary that accurately captures the main points of the legal opinion.Our approach involves using argument role information to generate multiple candidate summaries, then reranking these candidates based on alignment with the document's argument structure.We demonstrate the effectiveness of our approach on a dataset of long legal opinions and show that it outperforms several strong baselines.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation

In this paper, we define a widely neglected property in dialogue text, duality, which is a hierarchical property that is reflected in human behaviours in daily conversations: Based on the logic in a conversation (or a sentence), people can infer follow-up utterances (or tokens) based on the previous text, and vice versa.We propose a hierarchical duality learning for dialogue (HDLD) to simulate this human cognitive ability, for generating high quality responses that connect both previous and follow-up dialogues.HDLD utilizes hierarchical dualities at token hierarchy and utterance hierarchy.HDLD maximizes the mutual information between past and future utterances.Thus, even if the future text is invisible during inference, HDLD is capable of estimating future information implicitly based on dialogue history and generates both coherent and informative responses.In contrast to previous approaches that solely utilize future text as auxiliary information to encode during training, HDLD leverages duality to enable interaction between dialogue history and the future.This enhances the utilization of dialogue data, leading to the improvement in both automatic and human evaluation.
Retrieval-Augmented Domain Adaptation of Language Models

Language models pretrained on general domain corpora usually exhibit considerable degradation when generalizing to downstream tasks of specialized domains.Existing approaches try to construct PLMs for each specific domains either from scratch or through further pretraining, which not only costs substantial resources, but also fails to cover all target domains at various granularity.In this work, we propose RADA, a novel Retrieval-Augmented framework for Domain Adaptation.We first construct a textual corpora that covers the downstream task at flexible domain granularity and resource availability.We employ it as a pluggable datastore to retrieve informative background knowledge, and integrate them into the standard language model framework to augment representations.We then propose a two-level selection scheme to integrate the most relevant information while alleviating irrelevant noises.Specifically, we introduce a differentiable sampling module as well as an attention mechanism to achieve both passage-level and word-level selection.Such a retrieval-augmented framework enables domain adaptation of language models with flexible domain coverage and fine-grained domain knowledge integration.We conduct comprehensive experiments across biomedical, science and legal domains to demonstrate the effectiveness of the overall framework, and its advantage over existing solutions.
Medical Dialogue Generation via Dual Flow Modeling

Medical dialogue systems (MDS) aim to provide patients with medical services, such as diagnosis and prescription.Since most patients cannot precisely describe their symptoms, dialogue understanding is challenging for MDS.Previous studies mainly addressed this by extracting the mentioned medical entities as critical dialogue history information.In this work, we argue that it is also essential to capture the transitions of the medical entities and the doctor's dialogue acts in each turn, as they help the understanding of how the dialogue flows and enhance the prediction of the entities and dialogue acts to be adopted in the following turn.Correspondingly, we propose a Dual Flow enhanced Medical (DFMED) dialogue generation framework.It extracts the medical entities and dialogue acts used in the dialogue history and models their transitions with an entity-centric graph flow and a sequential act flow, respectively.We employ two sequential models to encode them and devise an interweaving component to enhance their interactions.Experiments on two datasets demonstrate that our method exceeds baselines in both automatic and manual evaluations.* Equal Contributions. 1) Pa#ent: I had a stomachache in the a+ernoon the day before yesterday and then started vomi#ng and had diarrhoea at night.A+er drinking four tubes of Huoxiang Zhengqi water, I s8ll vomit and can't eat.2) Doctor: Hello, I am glad to answer your ques8ons!Where exactly is the stomachache?⚕ Act Flow … 8) Doctor: You may have acute gastroenteri#s.It is recommended to take Omeprazole, Mosapride and Bifidobacterium tablets.
OASum: Large-Scale Open Domain Aspect-based Summarization

Aspect or query-based summarization has recently caught more attention, as it can generate differentiated summaries based on users' interests.However, the current dataset for aspect or query-based summarization either focuses on specific domains, contains relatively small-scale instances, or includes only a few aspect types.Such limitations hinder further explorations in this direction.In this work, we take advantage of crowd-sourcing knowledge on Wikipedia.organd automatically create a high-quality, large-scale open-domain aspectbased summarization dataset named OASum, which contains more than 3.7 million instances with around 1 million different aspects on 2 million Wikipedia pages.We provide benchmark results on OASum and demonstrate its ability for diverse aspect-based summarization generation.To overcome the data scarcity problem on specific domains, we also perform zero-shot, few-shot, and fine-tuning on seven downstream datasets.Specifically, zero/few-shot and finetuning results show that the model pre-trained on our corpus demonstrates a strong aspect or query-focused generation ability compared with the backbone model.Our dataset and pretrained checkpoints are publicly available.
DSPM-NLG: A Dual Supervised Pre-trained Model for Few-shot Natural Language Generation in Task-oriented Dialogue System

In few-shot settings, fully conveying the semantic information of dialogue act is a crucial challenge for Natural Language Generation (NLG) in task-oriented dialogue systems.It is noteworthy that NLG and Spoken Language Understanding (SLU) form a natural dual problem pair.If the SLU module can successfully restore the generated response by the NLG module to the corresponding dialogue act, this would demonstrate that the response is effectively conveying the semantic information of the dialogue act.Based on this idea, a novel Dual Supervised Pre-trained Model for a few-shot Natural Language Generation (DSPM-NLG) is proposed to regularize the pre-training process.We adopt a joint model with a dual supervised framework to learn the dual correlation between NLG and SLU from a probabilistic perspective.In addition, a slot-masked strategy is designed to enable the model to focus more effectively on the key slot-value pairs.DSPM-NLG is continuously trained on publicly available and large-scale labeled data, allowing it to gain a thorough understanding of the duality between the two tasks and to enhance the pre-trained model's ability for semantic control and generalization.Experimental results illustrate that our proposed model demonstrates exceptional performance on the few-shot benchmark dataset, outperforming the previous state-of-the-art results.
Utterance Classification with Logical Neural Network: Explainable AI for Mental Disorder Diagnosis

In response to the global challenge of mental health problems, we proposes a Logical Neural Network (LNN) based Neuro-Symbolic AI method for the diagnosis of mental disorders.Due to the lack of effective therapy coverage for mental disorders, there is a need for an AI solution that can assist therapists with the diagnosis.However, current Neural Network models lack explainability and may not be trusted by therapists.The LNN is a Recurrent Neural Network architecture that combines the learning capabilities of neural networks with the reasoning capabilities of classical logic-based AI.The proposed system uses input predicates from clinical interviews to output a mental disorder class, and different predicate pruning techniques are used to achieve scalability and higher scores.In addition, we provide an insight extraction method to aid therapists with their diagnosis.The proposed system addresses the lack of explainability of current Neural Network models and provides a more trustworthy solution for mental disorder diagnosis.
On the Interactions of Structural Constraints and Data Resources for Structured Prediction

In this work, we provide an analysis on the interactions of the effectiveness of decoding with structural constraints and the amount of available training data for structured prediction tasks in NLP.Our exploration adopts a simple protocol that enforces constraints upon constraint-agnostic local models at testing time.With evaluations on three typical structured prediction tasks (named entity recognition, dependency parsing, and event argument extraction), we find that models trained with less data predict outputs with more structural violations in greedy decoding mode.Incorporating constraints provides consistent performance improvements and such benefits are larger in lower resource scenarios.Moreover, there are similar patterns with regard to the model sizes and more efficient models tend to enjoy more benefits.Finally, we also investigate settings with genre transfer and discover patterns that are related to domain discrepancies.
infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information

The success of NLP systems often relies on the availability of large, high-quality datasets.However, not all samples in these datasets are equally valuable for learning, as some may be redundant or noisy.Several methods for characterizing datasets based on modeldriven meta-information (e.g., model's confidence) have been developed, but the relationship and complementary effects of these methods have received less attention.In this paper, we introduce infoVerse, a universal framework for dataset characterization, which provides a new feature space that effectively captures multidimensional characteristics of datasets by incorporating various model-driven metainformation.infoVerse reveals distinctive regions of the dataset that are not apparent in the original semantic space, hence guiding users (or models) in identifying which samples to focus on for exploration, assessment, or annotation.Additionally, we propose a novel sampling method on infoVerse to select a set of data points that maximizes informativeness.In three real-world applications (data pruning, active learning, and data annotation), the samples chosen on infoVerse space consistently outperform strong baselines in all applications.Our code and demo are publicly available.
HITS at DISRPT 2023: Discourse Segmentation, Connective Detection, and Relation Classification

HITS participated in the Discourse Segmentation (DS, Task 1) and Connective Detection (CD, Task 2) tasks at the DISRPT 2023.Task 1 focuses on segmenting the text into discourse units, while Task 2 aims to detect the discourse connectives.We deployed a framework based on different pre-trained models according to the target language for these two tasks.HITS also participated in the Relation Classification track (Task 3).The main task was recognizing the discourse relation between text spans from different languages.We designed a joint model for languages with a small corpus while separate models for large corpora.The adversarial training strategy is applied to enhance the robustness of relation classifiers.The implementation of our models for three tasks is available at https://github.com/liuwei1206/disrpt2023. * Equal contribution.Wei is responsible for relation classification, while Yi works on discourse segmentation and connective detection.
Dual Cache for Long Document Neural Coreference Resolution

Recent works show the effectiveness of cachebased neural coreference resolution models on long documents.These models incrementally process a long document from left to right and extract relations between mentions and entities in a cache, resulting in much lower memory and computation cost compared to computing all mentions in parallel.However, they do not handle cache misses when high-quality entities are purged from the cache, which causes wrong assignments and leads to prediction errors.We propose a new hybrid cache that integrates two eviction policies to capture global and local entities separately, and effectively reduces the aggregated cache misses up to half as before, while improving F1 score of coreference by 0.7 ∼ 5.7pt.As such, the hybrid policy can accelerate existing cache-based models and offer a new long document coreference resolution solution.Results show that our method outperforms existing methods on four benchmarks while saving up to 83% of inference time against non-cache-based models.Further, we achieve a new state-of-the-art on a long document coreference benchmark, LitBank.
Duplex Diffusion Models Improve Speech-to-Speech Translation

Speech-to-speech translation is a typical sequence-to-sequence learning task that naturally has two directions. How to effectively leverage bidirectional supervision signals to produce high-fidelity audio for both directions? Existing approaches either train two separate models or a multitask-learned model with low efficiency and inferior performance. In this paper, we propose a duplex diffusion model that applies diffusion probabilistic models to both sides of a reversible duplex Conformer, so that either end can simultaneously input and output a distinct language’s speech. Our model enables reversible speech translation by simply flipping the input and output ends. Experiments show that our model achieves the first success of reversible speech translation with significant improvements of ASR-BLEU scores compared with a list of state-of-the-art baselines.
PCJ at SemEval-2023 Task 10: A Ensemble Model Based on Pre-trained Model for Sexism Detection and Classification in English

This paper describes the system and the resulting model submitted by our team "PCJ" to the SemEval-2023 Task 10 sub-task A contest.In this task, we need to test the English text content in the posts to determine whether there is sexism, which involves emotional text classification.Our submission system utilizes methods based on RoBERTa, SimCSE-RoBERTa pre-training models, and model ensemble to classify and train on datasets provided by the organizers.In the final assessment, our submission achieved a macro average F1 score of 0.8449, ranking 28th out of 84 teams in Task A.
Separating Context and Pattern: Learning Disentangled Sentence Representations for Low-Resource Extractive Summarization

Extractive summarization aims to select a set of salient sentences from the source document to form a summary.Context information has been considered one of the key factors for this task.Meanwhile, there also exist other pattern factors that can identify sentence importance, such as sentence position or certain n-gram tokens.However, such pattern information is only effective in specific datasets or domains and can not be generalized like the context information when there only exists limited data.In this case, current extractive summarization models may suffer from a performance drop when transferring to a new dataset.In this paper, we attempt to apply disentangled representation learning on extractive summarization, and separate the two key factors for the task, context and pattern, for a better generalization ability in the lowresource setting.To achieve this, we propose two groups of losses for encoding and disentangling sentence representations into context representations and pattern representations.In this case, we can either use only the context information in the zero-shot setting or fine-tune the pattern information in the few-shot setting.Experimental results on three summarization datasets from different domains show the effectiveness of our proposed approach.
Lexical Translation Inconsistency-Aware Document-Level Translation Repair

Following the idea of “one translation per discourse”, in this paper we aim to improve translation consistency via document-level translation repair (DocRepair), i.e., automatic post-editing on translations of documents. To this end, we propose a lexical translation inconsistency-aware DocRepair to explicitly model translation inconsistency. First we locate the inconsistency in automatic translation. Then we provide translation candidates for those inconsistency. Finally, we propose lattice-like input to properly model inconsistent tokens and phrases and their candidates. Experimental results on three document-level translation datasets show that based on G-Transformer, a state-of-the-art document-to-document (Doc2Doc) translation model, our Doc2Doc DocRepair achieves significant improvement on translation quality in BLEU scores, but also greatly improves lexical translation consistency.