Reviewerly/exHarmony huggingface dataset:
Attribute Controlled Dialogue Prompting

Prompt-tuning has become an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks.However, both discrete prompting and continuous prompting assume fixed prompts for all data samples within a task, neglecting the fact that inputs vary greatly in some tasks such as open-domain dialogue generation.In this paper, we present a novel, instancespecific prompt-tuning algorithm for dialogue generation.Specifically, we generate prompts based on instance-level control code, rather than the conversation history, to explore their impact on controlled dialogue generation.Experiments on popular open-domain dialogue datasets, evaluated on both automated metrics and human evaluation, demonstrate that our method is superior to prompting baselines and comparable to fine-tuning with only 5%-6% of total parameters.* Work done during an internship at Huawei.overhead.We present results on both intent and persona controlled dialogue.2 Related Work GPT-3 (Brown et al., 2020) introduces prompting, a method to steer a frozen PLM by transforming inputs into cloze-style phrases with task description and some task examples.Though it is memoryefficient since one single copy of the PLM can be shared across different tasks, the model's performance is largely restricted by the maximum conditional input length, the model size and manual guesswork for prompts (
Multilingual Pre-training with Self-supervision from Global Co-occurrence Information

Global co-occurrence information is the primary source of structural information on multilingual corpora, and we find that analogical/parallel compound words across languages have similar co-occurrence counts/frequencies (normalized) giving weak but stable self-supervision for cross-lingual transfer. Following the observation, we aim at associating contextualized representations with relevant (contextualized) representations across languages with the help of co-occurrence counts. The result is MLM-GC (MLM with Global Co-occurrence) pre-training that the model learns local bidirectional information from MLM and global co-occurrence information from a log-bilinear regression. Experiments show that MLM-GC pre-training substantially outperforms MLM pre-training for 4 downstream cross-lingual tasks and 1 additional monolingual task, showing the advantages of forming isomorphic spaces across languages.
EM Pre-training for Multi-party Dialogue Response Generation

Dialogue response generation requires an agent to generate a response according to the current dialogue history, in terms of which twoparty dialogues have been well studied, but leaving a great gap for multi-party dialogues at the same time.Different from two-party dialogues where each response is a direct reply to its previous utterance, the addressee of a response utterance should be specified before it is generated in the multi-party scenario.Thanks to the huge amount of two-party conversational data, various pre-trained language models for two-party dialogue response generation have been proposed.However, due to the lack of annotated addressee labels in multi-party dialogue datasets, it is hard to use them to pre-train a response generation model for multi-party dialogues.To tackle this obstacle, we propose an Expectation-Maximization (EM) approach that iteratively performs the expectation steps to generate addressee labels, and the maximization steps to optimize a response generation model.Theoretical analyses and extensive experiments have justified the feasibility and effectiveness of our proposed method.The official implementation of this paper is available at https:
Constructing Word-Context-Coupled Space Aligned with Associative Knowledge Relations for Interpretable Language Modeling

As the foundation of current natural language processing methods, pre-trained language model has achieved excellent performance.However, the black-box structure of the deep neural network in pre-trained language models seriously limits the interpretability of the language modeling process.After revisiting the coupled requirement of deep neural representation and semantics logic of language modeling, a Word-Context-Coupled Space (W2CSpace) is proposed by introducing the alignment processing between uninterpretable neural representation and interpretable statistical logic.Moreover, a clustering process is also designed to connect the word-and context-level semantics.Specifically, an associative knowledge network (AKN), considered interpretable statistical logic, is introduced in the alignment process for word-level semantics.Furthermore, the context-relative distance is employed as the semantic feature for the downstream classifier, which is greatly different from the current uninterpretable semantic representations of pre-trained models.Our experiments for performance evaluation and interpretable analysis are executed on several types of datasets, including SIGHAN, Weibo, and ChnSenti.Wherein a novel evaluation strategy for the interpretability of machine learning models is first proposed.According to the experimental results, our language model can achieve better performance and highly credible interpretable ability compared to related stateof-the-art methods.
Hybrid Hierarchical Retrieval for Open-Domain Question Answering

Retrieval accuracy is crucial to the performance of open-domain question answering (ODQA) systems.Recent work has demonstrated that dense hierarchical retrieval (DHR), which retrieves document candidates first and then relevant passages from the refined document set, can significantly outperform the single stage dense passage retriever (DPR).While effective, this approach requires document structure information to learn document representation and is hard to adopt to other domains without this information.Additionally, the dense retrievers tend to generalize poorly on out-of-domain data comparing with sparse retrievers such as BM25.In this paper, we propose Hybrid Hierarchical Retrieval (HHR) to address the existing limitations.Instead of relying solely on dense retrievers, we can apply sparse retriever, dense retriever, and a combination of them in both stages of document and passage retrieval.We perform extensive experiments on ODQA benchmarks and observe that our framework not only brings in-domain gains, but also generalizes better to zero-shot TriviaQA and Web Questions datasets with an average of 4.69% improvement on recall@100 over DHR.We also offer practical insights to trade off between retrieval accuracy, latency, and storage cost.The code is available on github * .
Composition-contrastive Learning for Sentence Embeddings

Vector representations of natural language are ubiquitous in search applications.Recently, various methods based on contrastive learning have been proposed to learn textual representations from unlabelled data; by maximizing alignment between minimally-perturbed embeddings of the same text, and encouraging a uniform distribution of embeddings across a broader corpus.Differently, we propose maximizing alignment between texts and a composition of their phrasal constituents.We consider several realizations of this objective and elaborate the impact on representations in each case.Experimental results on semantic textual similarity tasks show improvements over baselines that are comparable with state-of-the-art approaches.Moreover, this work is the first to do so without incurring costs in auxiliary training objectives or additional network parameters. 1
Typology Guided Multilingual Position Representations: Case on Dependency Parsing

Recent multilingual models benefit from strong unified semantic representation models.However, due to conflict linguistic regularities, ignoring language-specific features during multilingual learning may suffer from negative transfer.In this work, we analyze the relation between a language's position space and its typological characterization, and suggest deploying different position spaces for different languages.We develop a position generation network which combines prior knowledge from typology features and existing position vectors.Experiments on the multilingual dependency parsing task show that the learned position vectors exhibit meaningful hidden structures, and they can help achieving the best multilingual parsing results.
Are Synonym Substitution Attacks Really Synonym Substitution Attacks?

In this paper, we explore the following question: Are synonym substitution attacks really synonym substitution attacks (SSAs)?We approach this question by examining how SSAs replace words in the original sentence and show that there are still unresolved obstacles that make current SSAs generate invalid adversarial samples.We reveal that four widely used word substitution methods generate a large fraction of invalid substitution words that are ungrammatical or do not preserve the original sentence's semantics.Next, we show that the semantic and grammatical constraints used in SSAs for detecting invalid word replacements are highly insufficient in detecting invalid adversarial samples.
HULAT at SemEval-2023 Task 10: Data Augmentation for Pre-trained Transformers Applied to the Detection of Sexism in Social Media

This paper describes our participation in SemEval-2023 Task 10, whose goal is the detection of sexism in social media.We explore some of the most popular transformer models such as BERT, DistilBERT, RoBERTa, and XL-Net.We also study different data augmentation techniques to increase the training dataset.During the development phase, our best results were obtained by using RoBERTa and data augmentation for tasks B and C.However, the use of synthetic data does not improve the results for task A. We participated in the three subtasks.Our approach still has much room for improvement, especially in the two fine-grained classifications.
Language-Agnostic Transformers and Assessing ChatGPT-Based Query Rewriting for Multilingual Document-Grounded QA

The DialDoc 2023 shared task has expanded the document-grounded dialogue task to encompass multiple languages, despite having limited annotated data.This paper assesses the effectiveness of both language-agnostic and language-aware paradigms for multilingual pretrained transformer models in a bi-encoderbased dense passage retriever (DPR), concluding that the language-agnostic approach is superior.Additionally, the study investigates the impact of query rewriting techniques using large language models, such as ChatGPT, on multilingual, document-grounded questionanswering systems.The experiments conducted demonstrate that, for the examples examined, query rewriting does not enhance performance compared to the original queries.This failure is due to topic switching in final dialogue turns and irrelevant topics being considered for query rewriting.
Resolving Ambiguities in Text-to-Image Generative Models

Natural language often contains ambiguities that can lead to misinterpretation and miscommunication.While humans can handle ambiguities effectively by asking clarifying questions and/or relying on contextual cues and commonsense knowledge, resolving ambiguities can be notoriously hard for machines.In this work, we study ambiguities that arise in text-to-image generative models.We curate the Text-toimage Ambiguity Benchmark (TAB) dataset to study different types of ambiguities in text-toimage generative models. 1 We then propose the Text-to-ImagE Disambiguation (TIED) framework to disambiguate the prompts given to the text-to-image generative models by soliciting clarifications from the end user.Through automatic and human evaluations, we show the effectiveness of our framework in generating more faithful images aligned with end user intention in the presence of ambiguities.
Enhancing Spanish-Quechua Machine Translation with Pre-Trained Models and Diverse Data Sources: LCT-EHU at AmericasNLP Shared Task

We present the LCT-EHU submission to the AmericasNLP 2023 low-resource machine translation shared task.We focus on the Spanish-Quechua language pair and explore the usage of different approaches: (1) Obtain new parallel corpora from the literature and legal domains, (2) Compare a high-resource Spanish-English pre-trained MT model with a Spanish-Finnish pre-trained model (with Finnish being chosen as a target language due to its morphological similarity to Quechua), and (3) Explore additional techniques such as copied corpus and back-translation.Overall, we show that the Spanish-Finnish pre-trained model outperforms other setups, while low-quality synthetic data reduces the performance.
Sren Kierkegaard at SemEval-2023 Task 4: Label-aware text classification using Natural Language Inference

In this paper, we describe our approach to Task 4 in SemEval 2023. Our pipeline tries to solve the problem of multi-label text classification of human values in English-written arguments. We propose a label-aware system where we reframe the multi-label task into a binary task resembling an NLI task. We propose to include the semantic description of the human values by comparing each description to each argument and ask whether there is entailment or not.
To Revise or Not to Revise: Learning to Detect Improvable Claims for Argumentative Writing Support

Optimizing the phrasing of argumentative text is crucial in higher education and professional development.However, assessing whether and how the different claims in a text should be revised is a hard task, especially for novice writers.In this work, we explore the main challenges to identifying argumentative claims in need of specific revisions.By learning from collaborative editing behaviors in online debates, we seek to capture implicit revision patterns in order to develop approaches aimed at guiding writers in how to further improve their arguments.We systematically compare the ability of common word embedding models to capture the differences between different versions of the same text, and we analyze their impact on various types of writing issues.To deal with the noisy nature of revision-based corpora, we propose a new sampling strategy based on revision distance.Opposed to approaches from prior work, such sampling can be done without employing additional annotations and judgments.Moreover, we provide evidence that using contextual information and domain knowledge can further improve prediction results.How useful a certain type of context is, depends on the issue the claim is suffering from, though.
NLP_CHRISTINE at SemEval-2023 Task 10: Utilizing Transformer Contextual Representations and Ensemble Learning for Sexism Detection on Social Media Texts

The paper describes the SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS), which investigates the detection of sexism on two social media sites, Gab and Reddit, by encouraging the development of machine learning models that perform binary and multi-class classification on English texts.The EDOS Task consisted of three hierarchical sub-tasks: binary sexism detection in sub-task A, category of sexism detection in sub-task B and fine-grained vector of sexism detection in sub-task C. My participation in EDOS comprised fine-tuning of different layer representations of Transformer-based pre-trained language models, namely BERT, AlBERT and RoBERTa, and ensemble learning via majority voting of the best performing models.Despite the low rank mainly due to a submission error, the system employed the largest version of the aforementioned Transformer models (BERT-Large, ALBERT-XXLarge-v1, ALBERT-XXLarge-v2, RoBERTa-Large), experimented with their multi-layer structure and aggregated their predictions so as to get the final result.My predictions on the test sets achieved 82.88%, 63.77% and 43.08% Macro-F1 score in sub-tasks A, B and C respectively.
Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation

This paper studies the impact of layer normalization (LayerNorm) on zero-shot translation (ZST).Recent efforts for ZST often utilize the Transformer architecture as the backbone, with LayerNorm at the input of layers (PreNorm) set as the default.However, Xu et al. ( 2019) has revealed that PreNorm carries the risk of overfitting the training data.Based on this, we hypothesize that PreNorm may overfit supervised directions and thus have low generalizability for ZST.Through experiments on OPUS, IWSLT, and Europarl datasets for 54 ZST directions, we demonstrate that the original Transformer setting of LayerNorm after residual connections (PostNorm) consistently outperforms PreNorm by up to 12.3 BLEU points.We then study the performance disparities by analyzing the differences in off-target rates and structural variations between PreNorm and PostNorm.This study highlights the need for careful consideration of the LayerNorm setting for ZST.
HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation

Recent NLP models have shown the remarkable ability to effectively generalise 'zero-shot' to new tasks using only natural language instructions as guidance.However, many of these approaches suffer from high computational costs due to their reliance on concatenating lengthy instructions with every input example, resulting in costly reprocessing of the instruction.To avoid this, we introduce Hypernetworks for INstruction Tuning (HINT), which convert task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder, eliminating the need to include instructions in the model input.The hypernetwork in HINT also produces an encoded instruction, which we concatenate with encoded inputs during decoding to further improve performance.HINT models outperform strong state-of-theart baselines by over 10% when controlling for compute (measured in FLOPs).By converting instructions into modules, HINT models can effectively disregard the length of instructions and few-shot example inputs in terms of compute usage.As a result, HINT can enhance its performance by up to 25% by incorporating additional few-shot data, while utilizing only up to 5% more compute.This combines the strengths of parameter-efficient fine-tuning and in-context learning.We release our code publicly 1 .
On-the-fly Cross-lingual Masking for Multilingual Pre-training

In multilingual pre-training with the objective of MLM (masked language modeling) on multiple monolingual corpora, multilingual models only learn cross-linguality implicitly from isomorphic spaces formed by overlapping different language spaces due to the lack of explicit cross-lingual forward pass.In this work, we present CLPM (Cross-lingual Prototype Masking), a dynamic and token-wise masking scheme, for multilingual pre-training, using a special token [C] x to replace a random token x in the input sentence.[C] x is a cross-lingual prototype for x and then forms an explicit crosslingual forward pass.We instantiate CLPM for the multilingual pre-training phase of UNMT (unsupervised neural machine translation), and experiments show that CLPM can consistently improve the performance of UNMT models on {De, Ro, N e} ↔ En.Beyond UNMT or bilingual tasks, we show that CLPM can consistently improve the performance of multilingual models on cross-lingual classification.
A Mutual Information-based Approach to Quantifying Logography in Japanese and Sumerian

Writing systems have traditionally been classified by whether they prioritize encoding phonological information (phonographic) versus morphological or semantic information (logographic).Recent work has broached the question of how membership in these categories can be quantified.We aim to contribute to this line of research by treating a definition of logography which directly incorporates morphological identity.Our methods compare mutual information between graphic forms and phonological forms and between graphic forms and morphological identity.We report on preliminary results here for two case studies, written Sumerian and written Japanese.The results suggest that our methods present a promising means of classifying the degree to which a writing system is logographic or phonographic.
SummQA at MEDIQA-Chat 2023: In-Context Learning with GPT-4 for Medical Summarization

Medical dialogue summarization is challenging due to the unstructured nature of medical conversations, the use of medical terminology in gold summaries, and the need to identify key information across multiple symptom sets.We present a novel system for the Dialogue2Note Medical Summarization tasks in the MEDIQA 2023 Shared Task.Our approach for sectionwise summarization (Task A) is a two-stage process of selecting semantically similar dialogues and using the top-k similar dialogues as in-context examples for GPT-4.For full-note summarization (Task B), we use a similar solution with k=1.We achieved 3rd place in Task A (2nd among all teams), 4th place in Task B Division Wise Summarization (2nd among all teams), 15th place in Task A Section Header Classification (9th among all teams), and 8th place among all teams in Task B. Our results highlight the effectiveness of few-shot prompting for this task, though we also identify several weaknesses of prompting-based approaches.We compare GPT-4 performance with several finetuned baselines.We find that GPT-4 summaries are more abstractive and shorter.We make our code publicly available 1 .
Meta-training with Demonstration Retrieval for Efficient Few-shot Learning

Large language models show impressive results on few-shot NLP tasks.However, these models are memory and computation-intensive.Metatraining allows one to leverage smaller models for few-shot generalization in a domaingeneral and task-agnostic manner (Min et al., 2022a;Wei et al., 2022;Chen et al., 2022); however, these methods alone results in models that may not have sufficient parameterization or knowledge to adapt quickly to a large variety of tasks.To overcome this issue, we propose meta-training with demonstration retrieval, where we use a dense passage retriever to retrieve semantically similar labeled demonstrations to each example for more varied supervision.By separating external knowledge from model parameters, we can use meta-training to train parameter-efficient models that generalize well on a larger variety of tasks.We construct a meta-training set from UNIFIEDQA and CROSSFIT, and propose a demonstration bank based on UNIFIEDQA tasks.To our knowledge, our work is the first to combine retrieval with meta-training, to use DPR models to retrieve demonstrations, and to leverage demonstrations from many tasks simultaneously, rather than randomly sampling demonstrations from the training set of the target task.Our approach outperforms a variety of targeted parameter-efficient and retrievalaugmented few-shot methods on QA, NLI, and text classification tasks (including SQuAD, QNLI, and TREC).Our approach can be metatrained and fine-tuned quickly on a single GPU.