Reviewerly/exHarmony huggingface dataset:
PeerDA: Data Augmentation via Modeling Peer Relation for Span Identification Tasks

Span identification aims at identifying specific text spans from text input and classifying them into pre-defined categories.Different from previous works that merely leverage the Subordinate (SUB) relation (i.e. if a span is an instance of a certain category) to train models, this paper for the first time explores the Peer (PR) relation, which indicates that two spans are instances of the same category and share similar features.Specifically, a novel Peer Data Augmentation (PeerDA) approach is proposed which employs span pairs with the PR relation as the augmentation data for training.PeerDA has two unique advantages: (1) There are a large number of PR span pairs for augmenting the training data.(2) The augmented data can prevent the trained model from over-fitting the superficial span-category mapping by pushing the model to leverage the span semantics.Experimental results on ten datasets over four diverse tasks across seven domains demonstrate the effectiveness of PeerDA.Notably, PeerDA achieves state-of-the-art results on six of them. 1
One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks

Current multimodal models, aimed at solving Vision and Language (V+L) tasks, predominantly repurpose Vision Encoders (VE) as feature extractors.While many VEs-of different architectures, trained on different data and objectives-are publicly available, they are not designed for the downstream V+L tasks.Nonetheless, most current work assumes that a single pre-trained VE can serve as a generalpurpose encoder.In this work, we focus on analysis and aim to understand whether the information stored within different VEs is complementary, i.e. if providing the model with features from multiple VEs can improve the performance on a target task, and how they are combined.We exhaustively experiment with three popular VEs on six downstream V+L tasks and analyze the attention and VE-dropout patterns.Our analyses suggest that diverse VEs complement each other, resulting in improved downstream V+L task performance, where the improvements are not due to simple ensemble effects (i.e. the performance does not always improve when increasing the number of encoders).We demonstrate that future VEs, which are not repurposed, but explicitly designed for V+L tasks, have the potential of improving performance on the target V+L tasks.
A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition

Existing models for named entity recognition (NER) are mainly based on large-scale labeled datasets, which always obtain using crowdsourcing.However, it is hard to obtain a unified and correct label via majority voting from multiple annotators for NER due to the large labeling space and complexity of this task.To address this problem, we aim to utilize the original multi-annotator labels directly.Particularly, we propose a Confidence-based Partial Label Learning (CPLL) method to integrate the prior confidence (given by annotators) and posterior confidences (learned by models) for crowd-annotated NER.This model learns a token-and content-dependent confidence via an Expectation-Maximization (EM) algorithm by minimizing empirical risk.The true posterior estimator and confidence estimator perform iteratively to update the true posterior and confidence respectively.We conduct extensive experimental results on both real-world and synthetic datasets, which show that our model can improve performance effectively compared with strong baselines.
PAI at SemEval-2023 Task 2: A Universal System for Named Entity Recognition with External Entity Information

The MultiCoNER II task aims to detect complex, ambiguous, and fine-grained named entities in low-context situations and noisy scenarios like the presence of spelling mistakes and typos for multiple languages.The task poses significant challenges due to the scarcity of contextual information, the high granularity of the entities(up to 33 classes), and the interference of noisy data.To address these issues, our team PAI proposes a universal Named Entity Recognition (NER) system that integrates external entity information to improve performance.Specifically, our system retrieves entities with properties from the knowledge base (i.e.Wikipedia) for a given text, then concatenates entity information with the input sentence and feeds it into Transformer-based models.Finally, our system wins 2 first places, 4 second places, and 1 third place out of 13 tracks.The code is publicly available at https://github.com/diqiuzhuanzhuan/semeval-2023.
Automatic Creation of Named Entity Recognition Datasets by Querying Phrase Representations

Most weakly supervised named entity recognition (NER) models rely on domain-specific dictionaries provided by experts.This approach is infeasible in many domains where dictionaries do not exist.While a phrase retrieval model was used to construct pseudo-dictionaries with entities retrieved from Wikipedia automatically in a recent study, these dictionaries often have limited coverage because the retriever is likely to retrieve popular entities rather than rare ones.In this study, we present a novel framework, HighGEN, that generates NER datasets with high-coverage pseudo-dictionaries.Specifically, we create entity-rich dictionaries with a novel search method, called phrase embedding search, which encourages the retriever to search a space densely populated with various entities.In addition, we use a new verification process based on the embedding distance between candidate entity mentions and entity types to reduce the false-positive noise in weak labels generated by high-coverage dictionaries.We demonstrate that HighGEN outperforms the previous best model by an average F1 score of 4.7 across five NER benchmark datasets.
ALERT: Adapt Language Models to Reasoning Tasks

Ping Yu, Tianlu Wang, Olga Golovneva, Badr AlKhamissi, Siddharth Verma, Zhijing Jin, Gargi Ghosh, Mona Diab, Asli Celikyilmaz. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations

Pragmatic reasoning about another speaker's unspoken intent and state of mind is crucial to efficient and effective human communication.It is virtually omnipresent in conversations between humans, e.g., when someone asks "do you have a minute?",instead of interpreting it literally as a query about your schedule, you understand that the speaker might have requests that take time, and respond accordingly.In this paper, we present PRAGMATICQA, the first large-scale open-domain question answering (QA) dataset featuring 6873 QA pairs that explores pragmatic reasoning in conversations over a diverse set of topics.We designed innovative crowdsourcing mechanisms for interestbased and task-driven data collection to address the common issue of incentive misalignment between crowdworkers and potential users.To compare computational models' capability at pragmatic reasoning, we also propose several quantitative metrics to evaluate question answering systems on PRAGMATICQA.We find that state-of-the-art systems still struggle to perform human-like pragmatic reasoning, and highlight their limitations for future research.* These authors contributed equally.PQ designed the collection mechanism and evaluation metrics, and wrote the paper; ND carried out most of the groundwork for collection/experiments/analyses.CM/JH offered project guidance and support, and helped discuss experiments and analysis.
FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models

With increasing concerns about data privacy, there is an increasing necessity of fine-tuning pre-trained language models (PLMs) for adapting to downstream tasks located in end-user devices or local clients without transmitting data to the central server.This urgent necessity therefore calls the research of investigating federated learning (FL) for PLMs.However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system.To this end, we investigate the parameter-efficient tuning (PETuning) of PLMs and develop a corresponding federated benchmark for four representative PETuning methods, dubbed FedPETuning.Specifically, FedPETuning provides the first holistic empirical study of representative PLMs tuning methods in FL, covering privacy attacks, performance comparisons, and resource-constrained analysis.Intensive experimental results have indicated that FedPETuning can efficiently defend against privacy attacks and maintains acceptable performance with reducing heavy resource consumption.The open-source
The MineTrans Systems for IWSLT 2023 Offline Speech Translation and Speech-to-Speech Translation Tasks

This paper presents the MINETRANS Englishto-Chinese speech translation systems developed for two challenge tracks of IWSLT 2023: Offline Speech Translation (S2T) and Speechto-Speech Translation (S2ST).For the offline S2T track, MINETRANS employs a practical cascaded system consisting of automatic speech recognition (ASR) and machine translation (MT) modules to explore translation performance limits in both constrained and unconstrained settings.To this end, we investigate the effectiveness of multiple ASR architectures and two MT strategies, i.e., supervised in-domain fine-tuning and prompt-driven translation using ChatGPT.For the S2ST track, we propose a novel speech-to-unit translation (S2UT) framework to build an end-to-end system, which encodes the target speech as discrete units via our trained HuBERT and leverages the standard sequence-to-sequence model to learn the mapping between source speech and discrete units directly.We demonstrate that with a largescale dataset, such as 10,000 hours of training data, this approach can well handle the mapping without any auxiliary recognition tasks (i.e., ASR and MT tasks).To the best of our knowledge, we are the first and only one to successfully train and submit the end-to-end S2ST model on this challenging track.
Smart Word Suggestions for Writing Assistance

Enhancing word usage is a desired feature for writing assistance.To further advance research in this area, this paper introduces "Smart Word Suggestions" (SWS) task and benchmark.Unlike other works, SWS emphasizes end-to-end evaluation and presents a more realistic writing assistance scenario.This task involves identifying words or phrases that require improvement and providing substitution suggestions.The benchmark includes human-labeled data for testing, a large distantly supervised dataset for training, and the framework for evaluation.The test data includes 1,000 sentences written by English learners, accompanied by over 16,000 substitution suggestions annotated by 10 native speakers.The training dataset comprises over 3.7 million sentences and 12.7 million suggestions generated through rules.Our experiments with seven baselines demonstrate that SWS is a challenging task.Based on experimental analysis, we suggest potential directions for future research on SWS.The dataset and related codes is available at https://github.com/microsoft/SmartWordSuggestions.
Towards Open-Domain Twitter User Profile Inference

Twitter user profile inference utilizes information from Twitter to predict user attributes (e.g., occupation, location), which is controversial because of its usefulness for downstream applications and its potential to reveal users' privacy.Therefore, it is important for researchers to determine the extent of profiling in a safe environment to facilitate proper use and make the public aware of the potential risks.Contrary to existing approaches on limited attributes, we explore open-domain Twitter user profile inference.We conduct a case study where we collect publicly available WikiData public figure profiles and use diverse WikiData predicates for profile inference.After removing sensitive attributes, our data contains over 150K public figure profiles from WikiData, over 50 different attribute predicates, and over 700K attribute values.We further propose a promptbased generation method, which can infer values that are implicitly mentioned in the Twitter information.Experimental results show that the generation-based approach can infer more comprehensive user profiles than baseline extraction-based methods, but limitations still remain to be applied for real-world use.We also enclose a detailed ethical statement for our data, potential benefits and risks from this work, and our efforts to mitigate the risks. 1 * Equal contribution. 1 To prevent potential misuse of our resources, we will only release our resources for research purposes based on individual requests.Please see our ethics statement section for details and contact us directly for resource access.
CSIRO Data61 Team at BioLaySumm Task 1: Lay Summarisation of Biomedical Research Articles Using Generative Models

Lay summarisation aims at generating a summary for a non-expert audience which allows them to keep updated with the latest research in a specific field.Despite the significant advancements made in the field of text summarisation, lay summarisation remains relatively under-explored.We present a comprehensive set of experiments and analyses to investigate the effectiveness of existing pre-trained language models in generating lay summaries, focusing on the impact of two factors: model size and training data.When evaluating our models in BioLaySumm Shared Task, our submission ranked second for the relevance criteria and third overall among 21 competing teams.
Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data

This paper presents Structure Aware DeNse ReTrievAl (SANTA) model, which encodes user queries and structured data in one universal embedding space for retrieving structured data.SANTA proposes two pretraining methods to make language models structureaware and learn effective representations for structured data: 1) Structured Data Alignment, which utilizes the natural alignment relations between structured data and unstructured data for structure-aware pretraining.It contrastively trains language models to represent multi-modal text data and teaches models to distinguish matched structured data for unstructured texts.2) Masked Entity Prediction, which designs an entity-oriented mask strategy and asks language models to fill in the masked entities.Our experiments show that SANTA achieves state-of-the-art on code search and product search and conducts convincing results in the zero-shot setting.SANTA learns tailored representations for multi-modal text data by aligning structured and unstructured data pairs and capturing structural semantics by masking and predicting entities in the structured data.
Dual-Alignment Pre-training for Cross-lingual Sentence Embedding

Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding.However, our research indicates that token-level alignment is also crucial in multilingual scenarios, which has not been fully explored previously.Based on our findings, we propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding that incorporates both sentence-level and token-level alignment.To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart.This reconstruction objective encourages the model to embed translation information into the token representation.Compared to other token-level alignment methods such as translation language modeling, RTL is more suitable for dual encoder architectures and is computationally efficient.Extensive experiments on three sentencelevel cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding.
MoralDial: A Framework to Train and Evaluate Moral Dialogue Systems via Moral Discussions

Morality in dialogue systems has raised great attention in research recently.A moral dialogue system aligned with users' values could enhance conversation engagement and user connections.In this paper, we propose a framework, MORALDIAL to train and evaluate moral dialogue systems.In our framework, we first explore the communication mechanisms of morality and resolve expressed morality into three parts, which indicate the roadmap for building a moral dialogue system.Based on that, we design a simple yet effective method: constructing moral discussions between simulated specific users and the dialogue system.The constructed discussions consist of expressing, explaining, revising, and inferring moral views in dialogue exchanges, which makes conversational models learn morality well in a natural manner.Furthermore, we propose a novel evaluation method under the framework.We evaluate the multiple aspects of morality by judging the relation between dialogue responses and human values in discussions, where the multifaceted nature of morality is particularly considered.Automatic and manual experiments demonstrate that our framework is promising to train and evaluate moral dialogue systems.
Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings

Automatic Speech Recognition (ASR) in medical contexts has the potential to save time, cut costs, increase report accuracy, and reduce physician burnout.However, the healthcare industry has been slower to adopt this technology, in part due to the importance of avoiding medically-relevant transcription mistakes.In this work, we present the Clinical BERTScore (CBERTScore), an ASR metric that penalizes clinically-relevant mistakes more than others.We collect a benchmark of 18 clinician preferences on 149 realistic medical sentences called the Clinician Transcript Preference benchmark (CTP) and make it publicly available 1 for the community to further develop clinically-aware ASR metrics.To our knowledge, this is the first public dataset of its kind.We demonstrate that our metric more closely aligns with clinician preferences on medical sentences as compared to other metrics (WER, BLUE, METEOR, etc), sometimes by wide margins.
Generating Structured Pseudo Labels for Noise-resistant Zero-shot Video Sentence Localization

Video sentence localization aims to locate moments in an unstructured video according to a given natural language query.A main challenge is the expensive annotation costs and the annotation bias.In this work, we study video sentence localization in a zero-shot setting, which learns with only video data without any annotation.Existing zero-shot pipelines usually generate event proposals and then generate a pseudo query for each event proposal.However, their event proposals are obtained via visual feature clustering, which is query-independent and inaccurate; and the pseudo-queries are short or less interpretable.Moreover, existing approaches ignores the risk of pseudo-label noise when leveraging them in training.To address the above problems, we propose a Structurebased Pseudo Label generation (SPL), which first generate free-form interpretable pseudo queries before constructing query-dependent event proposals by modeling the event temporal structure.To mitigate the effect of pseudolabel noise, we propose a noise-resistant iterative method that repeatedly re-weight the training sample based on noise estimation to train a grounding model and correct pseudo labels.Experiments on the ActivityNet Captions and Charades-STA datasets demonstrate the advantages of our approach.
I run as fast as a rabbit, can you? A Multilingual Simile Dialogues Datasets

A simile is a figure of speech that compares two different things (called the tenor and the vehicle) via shared properties.The tenor and the vehicle are usually connected with comparator words such as "like" or "as".The simile phenomena are unique and complex in a real-life dialogue scene where the tenor and the vehicle can be verbal phrases or sentences, mentioned by different speakers, exist in different sentences, or occur in reversed order.However, the current simile research usually focuses on similes in a triplet tuple (tenor, property, vehicle) or a single sentence where the tenor and vehicle are usually entities or noun phrases, which could not reflect complex simile phenomena in real scenarios.In this paper, we propose a novel and high-quality multilingual simile dialogue (MSD) dataset to facilitate the study of complex simile phenomena.The MSD is the largest manually annotated simile data (∼20K) and it contains both English and Chinese data.Meanwhile, the MSD data can also be used on dialogue tasks to test the ability of dialogue systems when using similes.We design 3 simile tasks (recognition, interpretation, and generation) and 2 dialogue tasks (retrieval and generation) with MSD.For each task, we provide experimental results from strong pretrained or state-of-the-art models.The experiments demonstrate the challenge of MSD and we will release the data/code on GitHub.
Weighted Contrastive Learning With False Negative Control to Help Long-tailed Product Classification

Item categorization (IC) aims to classify product descriptions into leaf nodes in a categorical taxonomy, which is a key technology used in a wide range of applications.Along with the fact that most datasets often has a long-tailed distribution, classification performances on tail labels tend to be poor due to scarce supervision, causing many issues in real-life applications.To address IC task's long-tail issue, K-positive contrastive loss (KCL) is proposed on image classification task and can be applied on the IC task when using text-based contrastive learning, e.g., SimCSE.However, one shortcoming of using KCL has been neglected in previous research: false negative (FN) instances may harm the KCL's representation learning.To address the FN issue in the KCL, we proposed to reweight the positive pairs in the KCL loss with a regularization that the sum of weights should be constrained to K +1 as close as possible.After controlling FN instances with the proposed method, IC performance has been further improved and is superior to other LT-addressing methods.
Multi-Dialectal Representation Learning of Sinitic Phonology

Machine learning techniques have shown their competence for representing and reasoning in symbolic systems such as language and phonology.In Sinitic Historical Phonology, notable tasks that could benefit from machine learning include the comparison of dialects and reconstruction of proto-languages systems.Motivated by this, this paper provides an approach for obtaining multi-dialectal representations of Sinitic syllables, by constructing a knowledge graph from structured phonological data, then applying the BoxE technique from knowledge base learning.We applied unsupervised clustering techniques to the obtained representations to observe that the representations capture phonemic contrast from the input dialects.Furthermore, we trained classifiers to perform inference of unobserved Middle Chinese labels, showing the representations' potential for indicating archaic, proto-language features.The representations can be used for performing completion of fragmented Sinitic phonological knowledge bases, estimating divergences between different characters, or aiding the exploration and reconstruction of archaic features.