Reviewerly/exHarmony huggingface dataset:
Aspect-aware Unsupervised Extractive Opinion Summarization

Extractive opinion summarization extracts sentences from users' reviews to represent the prevalent opinions about a product or service.However, the extracted sentences can be redundant and may miss some important aspects, especially for centroid-based extractive summarization models (Radev et al., 2004).To alleviate these issues, we introduce TokenClustera method for unsupervised extractive opinion summarization that automatically identifies the aspects described in the review sentences and then extracts sentences based on their aspects.It identifies the underlying aspects of the review sentences using the roots of noun phrases and adjectives appearing in them.Empirical evaluation shows that TokenCluster improves aspect coverage in summaries and achieves strong performance on multiple opinion summarization datasets, for both general and aspectspecific summarization.We also perform extensive ablation and human evaluation studies to validate the design choices of our method.The implementation of our work is available at https://github.com/leehaoyuan/TokenCluster.
Soft Alignment Objectives for Robust Adaptation of Language Generation

Domain adaptation allows generative language models to address specific flaws caused by the domain shift of their application.However, the traditional adaptation by further training on indomain data rapidly weakens the model's ability to generalize to other domains, making the openended deployments of the adapted models prone to errors.This work introduces novel training objectives built upon a semantic similarity of the predicted tokens to the reference.Our results show that (1) avoiding the common assumption of a single correct prediction by constructing the training target from tokens' semantic similarity can largely mitigate catastrophic forgetting of adaptation, while (2) preserving the adaptation in-domain quality, (3) with negligible additions to compute costs.In the broader context, the objectives grounded in a continuous token similarity pioneer the exploration of the middle ground between the efficient but naïve exact-match token-level objectives and expressive but computationally-and resourceintensive sequential objectives.
CAISA at SemEval-2023 Task 8: Counterfactual Data Augmentation for Mitigating Class Imbalance in Causal Claim Identification

The class imbalance problem can cause machine learning models to produce an undesirable performance on the minority class as well as the whole dataset.Using data augmentation techniques to increase the number of samples is one way to tackle this problem.We introduce a novel counterfactual data augmentation by verb replacement for the identification of medical claims.In addition, we investigate the impact of this method and compare it with 3 other data augmentation techniques, showing that the proposed method can result in a significant (relative) improvement in the minority class.
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers

Intermediate training of pre-trained transformer-based language models on domainspecific data leads to substantial gains for downstream tasks.To increase efficiency and prevent catastrophic forgetting alleviated from full domain-adaptive pre-training, approaches such as adapters have been developed.However, these require additional parameters for each layer, and are criticized for their limited expressiveness.In this work, we introduce TADA, a novel task-agnostic domain adaptation method which is modular, parameter-efficient, and thus, data-efficient.Within TADA, we retrain the embeddings to learn domain-aware input representations and tokenizers for the transformer encoder, while freezing all other parameters of the model.Then, task-specific fine-tuning is performed.We further conduct experiments with meta-embeddings and newly introduced metatokenizers, resulting in one model per task in multi-domain use cases.Our broad evaluation in 4 downstream tasks for 14 domains across single-and multi-domain setups and high-and low-resource scenarios reveals that TADA is an effective and efficient alternative to full domainadaptive pre-training and adapters for domain adaptation, while not introducing additional parameters or complex training steps.
Multi-Dimensional Evaluation of Text Summarization with In-Context Learning

Evaluation of natural language generation (NLG) is complex and multi-dimensional.Generated text can be evaluated for fluency, coherence, factuality, or any other dimensions of interest.Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets.In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning, obviating the need for large training datasets.Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization, establishing state-of-the-art on dimensions such as relevance and factual consistency.We then analyze the effects of factors such as the selection and number of incontext examples on performance.Finally, we study the efficacy of in-context learningbased evaluators in evaluating zero-shot summaries written by large language models such as GPT-3.Our code is available at https: //github.com/JainSameer06/ICE
Exploring a New Grammatico-functional Type of Measure as Part of a Language Learning Expert System

This paper explores the use of L2-specific grammatical microsystems as elements of the domain knowledge of an Intelligent Computer-assisted Language Learning (ICALL) system.We report on the design of new grammatico-functional measures and their association with proficiency.We illustrate the approach with the design of the IT, THIS, THAT proform microsystem.The measures rely on the paradigmatic relations between words of the same linguistic functions.They are operationalised with one frequency-based and two probabilistic methods, i.e., the relative proportions of the forms and their likelihood of occurrence.Ordinal regression models show that the measures are significant in terms of association with CEFR levels, paving the way for their introduction in a specific proform microsystem expert model.
WizMap: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddings

Fig.
Hierarchical Verbalizer for Few-Shot Hierarchical Text Classification

Due to the complex label hierarchy and intensive labeling cost in practice, the hierarchical text classification (HTC) suffers a poor performance especially when low-resource or fewshot settings are considered.Recently, there is a growing trend of applying prompts on pretrained language models (PLMs), which has exhibited effectiveness in the few-shot flat text classification tasks.However, limited work has studied the paradigm of prompt-based learning in the HTC problem when the training data is extremely scarce.In this work, we define a path-based few-shot setting and establish a strict path-based evaluation metric to further explore few-shot HTC tasks.To address the issue, we propose the hierarchical verbalizer ("Hi-erVerb"), a multi-verbalizer framework treating HTC as a single-or multi-label classification problem at multiple layers and learning vectors as verbalizers constrained by hierarchical structure and hierarchical contrastive learning.In this manner, HierVerb fuses label hierarchy knowledge into verbalizers and remarkably outperforms those who inject hierarchy through graph encoders, maximizing the benefits of PLMs.Extensive experiments on three popular HTC datasets under the few-shot settings demonstrate that prompt with HierVerb significantly boosts the HTC performance, meanwhile indicating an elegant way to bridge the gap between the large pre-trained model and downstream hierarchical classification tasks.
TimelineQA: A Benchmark for Question Answering over Timelines

Lifelogs are descriptions of experiences that a person had during their life.Lifelogs are created by fusing data from the multitude of digital services, such as online photos, maps, shopping and content streaming services.Question answering over lifelogs can offer personal assistants a critical resource when they try to provide advice in context.However, obtaining answers to questions over lifelogs is beyond the current state of the art of question answering techniques for a variety of reasons, the most pronounced of which is that lifelogs combine free text with some degree of structure such as temporal and geographical information.We create and publicly release TimelineQA 1 , a benchmark for accelerating progress on querying lifelogs.TimelineQA generates lifelogs of imaginary people.The episodes in the lifelog range from major life episodes such as high school graduation to those that occur on a daily basis such as going for a run.We describe a set of experiments on TimelineQA with several state-of-the-art QA models.Our experiments reveal that for atomic queries, an extractive QA system significantly out-performs a state-of-the-art retrieval-augmented QA system.For multi-hop queries involving aggregates, we show that the best result is obtained with a stateof-the-art table QA technique, assuming the ground truth set of episodes for deriving the answer is available.
Misleading Relation Classifiers by Substituting Words in Texts

Relation classification is to determine the semantic relationship between two entities in a given sentence.However, many relation classifiers are vulnerable to adversarial attacks, which is using adversarial examples to lead victim models to output wrong results.In this paper, we propose a simple but effective method for misleading relation classifiers.We first analyze the most important parts of speech (POSs) from the syntax and morphology perspectives, then we substitute words labeled with these POS tags in original samples with synonyms or hyponyms.Experimental results show that our method can generate adversarial texts of high quality, and most of the relationships between entities can be correctly identified in the process of human evaluation.Furthermore, the adversarial examples generated by our method possess promising transferability, and they are also helpful for improving the robustness of victim models.
GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization

Automatic summarization with pre-trained language models has led to impressively fluent results, but is prone to ‘hallucinations’, low performance on non-news genres, and outputs which are not exactly summaries. Targeting ACL 2023’s ‘Reality Check’ theme, we present GUMSum, a small but carefully crafted dataset of English summaries in 12 written and spoken genres for evaluation of abstractive summarization. Summaries are highly constrained, focusing on substitutive potential, factuality, and faithfulness. We present guidelines and evaluate human agreement as well as subjective judgments on recent system outputs, comparing general-domain untuned approaches, a fine-tuned one, and a prompt-based approach, to human performance. Results show that while GPT3 achieves impressive scores, it still underperforms humans, with varying quality across genres. Human judgments reveal different types of errors in supervised, prompted, and human-generated summaries, shedding light on the challenges of producing a good summary.
Are Sample-Efficient NLP Models More Robust?

Recent results in image classification and extractive question answering have observed that pre-trained models trained on less in-distribution data have better out-ofdistribution performance.However, it is unclear how broadly these trends hold.We conduct a large empirical study across three tasks, three broadly-applicable modeling interventions (increasing model size, using a different adaptation method, and pre-training on more data), and 14 diverse datasets to investigate the relationship between sample efficiency (amount of data needed to reach a given ID accuracy) and robustness (how models fare on OOD evaluation).We find that higher sample efficiency is only correlated with better average OOD robustness on some modeling interventions and tasks, but not others.On individual datasets, models with lower sample efficiency can even be more robust.These results suggest that general-purpose methods for improving sample efficiency are unlikely to yield universal OOD robustness improvements, since such improvements are highly dataset-and task-dependent.Even in an era of large, multi-purpose pre-trained models, task-specific decisions may often be necessary for OOD generalization.# ID Examples Used ID Performance Model B has higher sample efficiency than Model A. Example Sample Efficiency Plot ID Performance OOD Performance Model B has higher effective robustness than Model A. Example Effective Robustness Plot Model A Model B
lasigeBioTM at SemEval-2023 Task 7: Improving Natural Language Inference Baseline Systems with Domain Ontologies

Clinical Trials Reports (CTRs) contain highly valuable health information from which Natural Language Inference (NLI) techniques determine if a given hypothesis can be inferred from a given premise.CTRs are abundant with domain terminology with particular terms that are difficult to understand without prior knowledge.Thus, we proposed to use domain ontologies as a source of external knowledge that could help with the inference process in the SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT).This document describes our participation in subtask 1: Textual Entailment, where Ontologies, NLP techniques, such as tokenization and namedentity recognition, and rule-based approaches are all combined in our approach.We were able to show that inputting annotations from domain ontologies improved the baseline systems.
e-Health CSIRO at RadSum23: Adapting a Chest X-Ray Report Generator to Multimodal Radiology Report Summarisation

We describe the participation of team e-Health CSIRO in the BioNLP RadSum task of 2023.This task aims to develop automatic summarisation methods for radiology.The subtask that we participated in was multimodal; the impression section of a report was to be summarised from a given findings section and set of Chest X-rays (CXRs) of a subject's study.For our method, we adapted an encoder-to-decoder model for CXR report generation to the subtask.e-Health CSIRO placed seventh amongst the participating teams with a RadGraph ER F1 score of 23.9.
Hybrid and Collaborative Passage Reranking

In passage retrieval system, the initial passage retrieval results may be unsatisfactory, which can be refined by a reranking scheme.Existing solutions to passage reranking focus on enriching the interaction between query and each passage separately, neglecting the context among the top-ranked passages in the initial retrieval list.To tackle this problem, we propose a Hybrid and Collaborative Passage Reranking (HybRank) method, which leverages the substantial similarity measurements of upstream retrievers for passage collaboration and incorporates the lexical and semantic properties of sparse and dense retrievers for reranking.Besides, built on off-the-shelf retriever features, HybRank is a plug-in reranker capable of enhancing arbitrary passage lists including previously reranked ones.Extensive experiments demonstrate the stable improvements of performance over prevalent retrieval and reranking methods, and verify the effectiveness of the core components of HybRank. 1 E ′ * = [e ′ * 1 , e ′ * 2 , . . ., e ′ * L ] and enhanced with the importance information of anchor passages.
Team JUSTR00 at SemEval-2023 Task 3: Transformers for News Articles Classification

The SemEval-2023 Task 3 competition offers participants a multi-lingual dataset annotated into three schemes, one for each subtask.The competition challenges participants to construct machine learning systems that can categorize news articles based on their nature and style of writing.We experiment with many state-of-the-art transformer-based language models proposed in the natural language processing literature and report the results of the best ones.Our best performing model is based on a transformer called "Longformer" and has achieved an F1-Micro score of 0.256 on the English version of subtask-1 and an F1-Macro score of 0.442 on subtask-2 on the test data.We also experiment with several state-ofthe-art multi-lingual transformer-based models and report the results of the best performing ones.
Mind the Gap between the Application Track and the Real World

Recent advances in NLP have led to a rise in inter-disciplinary and application-oriented research.While this demonstrates the growing real-world impact of the field, research papers frequently feature experiments that do not account for the complexities of realistic data and environments.To explore the extent of this gap, we investigate the relationship between the realworld motivations described in NLP papers and the models and evaluation which comprise the proposed solution.We first survey papers from the NLP Applications track from ACL 2020 and EMNLP 2020, asking which papers have differences between their stated motivation and their experimental setting, and if so, mention them.We find that many papers fall short of considering real-world input and output conditions due to adopting simplified modeling or evaluation settings.As a case study, we then empirically show that the performance of an educational dialog understanding system deteriorates when used in a realistic classroom environment.
Why Did the Chicken Cross the Road? Rephrasing and Analyzing Ambiguous Questions in VQA

Natural language is ambiguous.Resolving ambiguous questions is key to successfully answering them.Focusing on questions about images, we create a dataset of ambiguous examples.We annotate these, grouping answers by the underlying question they address and rephrasing the question for each group to reduce ambiguity.Our analysis reveals a linguistically-aligned ontology of reasons for ambiguity in visual questions.We then develop an English questiongeneration model which we demonstrate via automatic and human evaluation produces less ambiguous questions.We further show that the question generation objective we use allows the model to integrate answer group information without any direct supervision. 1
MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction

Given a query, the task of Natural Language Video Localization (NLVL) is to localize a temporal moment in an untrimmed video that semantically matches the query.In this paper, we adopt a proposal-based solution that generates proposals (i.e., candidate moments) and then select the best matching proposal.On top of modeling the cross-modal interaction between candidate moments and the query, our proposed Moment Sampling DETR (MS-DETR) enables efficient moment-moment relation modeling.The core idea is to sample a subset of moments guided by the learnable templates with an adopted DETR (DEtection TRansformer) framework.To achieve this, we design a multiscale visual-linguistic encoder, and an anchorguided moment decoder paired with a set of learnable templates.Experimental results on three public datasets demonstrate the superior performance of MS-DETR.
Can Cross-Lingual Transferability of Multilingual Transformers Be Activated Without End-Task Data?

Pretrained multilingual Transformers have achieved great success in cross-lingual transfer learning.Current methods typically activate the cross-lingual transferability of multilingual Transformers by fine-tuning them on end-task data.However, the methods cannot perform cross-lingual transfer when end-task data are unavailable.In this work, we explore whether the cross-lingual transferability can be activated without end-task data.We propose a cross-lingual transfer method, named PLUGIN-X.PLUGIN-X disassembles monolingual and multilingual Transformers into submodules, and reassembles them to be the multilingual end-task model.After representation adaptation, PLUGIN-X finally performs crosslingual transfer in a plug-and-play style.Experimental results show that PLUGIN-X successfully activates the cross-lingual transferability of multilingual Transformers without accessing end-task data.Moreover, we analyze how the cross-model representation alignment affects the cross-lingual transferability.
Bring More Attention to Syntactic Symmetry for Automatic Postediting of High-Quality Machine Translations

Automatic postediting (APE) is an automated process to refine a given machine translation (MT).Recent findings present that existing APE systems are not good at handling highquality MTs even for a language pair with abundant data resources, English-German: the better the given MT is, the harder it is to decide what parts to edit and how to fix these errors.One possible solution to this problem is to instill deeper knowledge about the target language into the model.Thus, we propose a linguistically motivated method of regularization that is expected to enhance APE models' understanding of the target language: a loss function that encourages symmetric self-attention on the given MT.Our analysis of experimental results demonstrates that the proposed method helps improving the state-of-the-art architecture's APE quality for high-quality MTs.
KIT’s Multilingual Speech Translation System for IWSLT 2023

Many existing speech translation benchmarks focus on native-English speech in high-quality recording conditions, which often do not match the conditions in real-life use-cases.In this paper, we describe our speech translation system for the multilingual track of IWSLT 2023, which focuses on the translation of scientific conference talks.The test condition features accented input speech and terminology-dense contents.The tasks requires translation into 10 languages of varying amounts of resources.In absence of training data from the target domain, we use a retrieval-based approach (kNN-MT) for effective adaptation (+0.8 BLEU for speech translation).We also use adapters to easily integrate incremental training data from data augmentation, and show that it matches the performance of re-training.We observe that cascaded systems are more easily adaptable towards specific target domains, due to their separate modules.Our cascaded speech system outperforms its end-to-end counterpart on scientific talk translation, although their performance remains similar on TED talks.
Are Layout-Infused Language Models Robust to Layout Distribution Shifts? A Case Study with Scientific Documents

Recent work has shown that infusing layout features into language models (LMs) improves processing of visually-rich documents such as scientific papers.Layout-infused LMs are often evaluated on documents with familiar layout features (e.g., papers from the same publisher), but in practice models encounter documents with unfamiliar distributions of layout features, such as new combinations of text sizes and styles, or new spatial configurations of textual elements.In this work, we test whether layoutinfused LMs are robust to layout distribution shifts.As a case study, we use the task of scientific document structure recovery, segmenting a scientific paper into its structural categories (e.g., TITLE, CAPTION, REFERENCE).To emulate distribution shifts that occur in practice, we re-partition the GROTOAP2 dataset.We find that under layout distribution shifts model performance degrades by up to 20 F1.Simple training strategies, such as increasing training diversity, can reduce this degradation by over 35% relative F1; however, models fail to reach in-distribution performance in any tested out-of-distribution conditions.This work highlights the need to consider layout distribution shifts during model evaluation, and presents a methodology for conducting such evaluations.