Reviewerly/exHarmony huggingface dataset:
PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation

Image translation is a task that translates an image containing text in the source language to the target language.One major challenge with image translation is the modality gap between visual text inputs and textual inputs/outputs of machine translation (MT).In this paper, we propose PEIT, an end-to-end image translation framework that bridges the modality gap with pre-trained models.It is composed of four essential components: a visual encoder, a shared encoder-decoder backbone network, a vision-text representation aligner equipped with the shared encoder and a cross-modal regularizer stacked over the shared decoder.Both the aligner and regularizer aim at reducing the modality gap.To train PEIT, we employ a twostage pre-training strategy with an auxiliary MT task: (1) pre-training the MT model on the MT training data to initialize the shared encoder-decoder backbone network; and (2) pre-training PEIT with the aligner and regularizer on a synthesized dataset with rendered images containing text from the MT training data.In order to facilitate the evaluation of PEIT and promote research on image translation, we create a large-scale image translation corpus ECOIT containing 480K imagetranslation pairs via crowd-sourcing and manual post-editing from real-world images in the e-commerce domain.Experiments on the curated ECOIT benchmark dataset demonstrate that PEIT substantially outperforms both cascaded image translation systems (OCR+MT) and previous strong end-to-end image translation model, with fewer parameters and faster decoding speed.Codes are available at https: //github.com/lishangjie1/PEIT.
Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios

Current pre-trained language models have enabled remarkable improvements in downstream tasks, but it remains difficult to distinguish effects of statistical correlation from more systematic logical reasoning grounded on the understanding of real world.We tease these factors apart by leveraging counterfactual conditionals, which force language models to predict unusual consequences based on hypothetical propositions.We introduce a set of tests from psycholinguistic experiments, as well as larger-scale controlled datasets, to probe counterfactual predictions from five pre-trained language models.We find that models are consistently able to override real-world knowledge in counterfactual scenarios, and that this effect is more robust in case of stronger baseline world knowledge-however, we also find that for most models this effect appears largely to be driven by simple lexical cues.When we mitigate effects of both world knowledge and lexical cues to test knowledge of linguistic nuances of counterfactuals, we find that only GPT-3 shows sensitivity to these nuances, though this sensitivity is also non-trivially impacted by lexical associative factors. 1
IUST_NLP at SemEval-2023 Task 10: Explainable Detecting Sexism with Transformers and Task-adaptive Pretraining

This paper describes our system on SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS).This work aims to design an automatic system for detecting and classifying sexist content in online spaces.We propose a set of transformer-based pre-trained models with task-adaptive pretraining and ensemble learning.The main contributions of our system include analyzing the performance of different transformer-based pre-trained models and combining these models, as well as providing an efficient method using large amounts of unlabeled data for model adaptive pretraining.We have also explored several other strategies.On the test dataset, our system achieves F1-scores of 83%, 64%, and 47% on subtasks A, B, and C, respectively.
The Effect of Alignment Correction on Cross-Lingual Annotation Projection

Cross-lingual annotation projection is a practical method for improving performance on low resource structured prediction tasks.An important step in annotation projection is obtaining alignments between the source and target texts, which enables the mapping of annotations across the texts.By manually correcting automatically generated alignments, we examine the impact of alignment quality-automatic, manual, and mixed-on downstream performance for two information extraction tasks and quantify the trade-off between annotation effort and model performance.
Extrinsic Evaluation of Machine Translation Metrics

Automatic machine translation (MT) metrics are widely used to distinguish the quality of machine translation systems across large test sets (i.e., system-level evaluation).However, it is unclear if automatic metrics can reliably distinguish good translations from bad at the sentence level (i.e., segment-level evaluation).We investigate how useful MT metrics are at detecting segment-level quality by correlating metrics with the translation utility for downstream tasks.We evaluate the segment-level performance of widespread MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks (dialogue state tracking, question answering, and semantic parsing).For each task, we have access to a monolingual task-specific model and a translation model.We calculate the correlation between the metric's ability to predict a good/bad translation with the success/failure on the final task for machine-translated test sentences.Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of downstream outcomes.We also find that the scores provided by neural metrics are not interpretable, in large part due to having undefined ranges.We synthesise our analysis into recommendations for future MT metrics to produce labels rather than scores for more informative interaction between machine translation and multilingual language understanding.
Legal_try at SemEval-2023 Task 6: Voting Heterogeneous Models for Entities identification in Legal Documents

Named Entity Recognition (NER) is a crucial subtask of Natural Language Processing (NLP) that involves identifying and categorizing named entities in text.The resulting annotation enables unstructured natural language texts for various NLP tasks, such as information retrieval, question answering, and machine translation.NER is essential in the legal domain as an initial stage for extracting relevant entities.However, legal texts contain domainspecific named entities, including applicants, defendants, courts, statutes, and articles, rendering standard named entity recognizers incompatible with legal documents.This paper proposes an approach that combines multiple model results through a voting mechanism to identify unique entities in legal texts.This study's primary focus is extracting named entities from legal texts in the context of SemEval-2023 Task 6, Sub-task B: Legal Named Entities Extraction (L-NER).The goal is to create a legal NER system for unique entity annotation in legal documents.Our experiments' results and our system's implementation are published and accessible online 1 .
Contextualized Semantic Distance between Highly Overlapped Texts

Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation.Better evaluation of the semantic distance between the overlapped sentences benefits the language system's understanding and guides the generation.Since conventional semantic metrics are based on word representations, they are vulnerable to the disturbance of overlapped components with similar representations.This paper aims to address the issue with a mask-and-predict strategy.We take the words in the longest common sequence (LCS) as neighboring words and use masked language modeling (MLM) from pre-trained language models (PLMs) to predict the distributions in their positions.Our metric, Neighboring Distribution Divergence (NDD), represents the semantic distance by calculating the divergence between distributions in the overlapped parts.Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.Based on the discovery, we further implement an unsupervised and training-free method for text compression, leading to a significant improvement on the previous perplexity-based method.The high compression rate controlling ability of our method even enables NDD to outperform the supervised state-of-the-art in domain adaption by a huge margin.Further experiments on syntax and semantics analyses verify the awareness of internal sentence structures, indicating the high potential of NDD for further studies.
SzegedAI at SemEval-2023 Task 1: Applying Quasi-Symbolic Representations in Visual Word Sense Disambiguation

In this paper, we introduce our submission in the task of visual word sense disambiguation (V-WSD).Our proposed solution operates by deriving quasi-symbolic semantic categories from the hidden representations of multi-modal text-image encoders.Our results are mixed, as we manage to achieve a substantial boost in performance when evaluating on a validation set, however, we experienced detrimental effects during evaluation on the actual test set.Our positive results on the validation set confirms the validity of the quasi-symbolic features, whereas our results on the test set revealed that the proposed technique was not able to cope with the sufficiently different distribution of the test data.
Auto-req: Automatic detection of pre-requisite dependencies between academic videos

Online learning platforms offer a wealth of educational material, but as the amount of content on these platforms grows, students may struggle to determine the most efficient order in which to cover the material to achieve a particular learning objective.In this paper, we propose a feature-based method for identifying pre-requisite dependencies between academic videos.Our approach involves using a transcript engine with a language model to transcribe domain-specific terms and then extracting novel similarity-based features to determine pre-requisite dependencies between video transcripts.This approach succeeds due to the development of a novel corpus of K-12 academic text, which was created using a proposed feature-based document parser.We evaluate our method on hand-annotated datasets for transcript extraction, video pre-requisites determination, and textbook parsing, which we have released.Our method for pre-requisite edge determination shows significant improvement (+4.7%-10.24%F1-score) compared to existing methods.
Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark

Recent model editing techniques promise to mitigate the problem of memorizing false or outdated associations during large language model (LLM) training.However, we show that these techniques can introduce large unwanted side effects which are not detected by existing specificity benchmarks.We extend the existing COUNTERFACT benchmark to include a dynamic component and dub our benchmark COUNTERFACT+.Additionally, we extend the metrics used for measuring specificity by a principled KL divergence-based metric.We use this improved benchmark to evaluate recent model editing techniques and find that they suffer from low specificity.Our findings highlight the need for improved specificity benchmarks that identify and prevent unwanted side effects.
Unified Syntactic Annotation of English in the CGEL Framework

We investigate whether the Cambridge Grammar of the English Language (2002) and its extensive descriptions work well as a corpus annotation scheme.We develop annotation guidelines and in the process outline some interesting linguistic uncertainties that we had to resolve.To test the applicability of CGEL to real-world corpora, we conduct an interannotator study on sentences from the English Web Treebank, showing that consistent annotation of even complex syntactic phenomena like gapping using the CGEL formalism is feasible.Why introduce yet another formalism for English syntax?We argue that CGEL is attractive due to its exhaustive analysis of English syntactic phenomena, its labeling of both constituents and functions, and its accessibility.We look towards expanding CGELBank and augmenting it with automatic conversions from existing treebanks in the future.
Deep Equilibrium Non-Autoregressive Sequence Learning

In this work, we argue that non-autoregressive (NAR) sequence generative models can equivalently be regarded as iterative refinement process towards the target sequence, implying an underlying dynamical system of NAR models: z = f (z, x) → y.In such a way, the optimal prediction of a NAR model should be the equilibrium state of its dynamics if given infinitely many iterations.However, this is infeasible in practice due to limited computational and memory budgets.To this end, we propose DEQNAR to directly solve for the equilibrium state of NAR models based on deep equilibrium networks (Bai et al., 2019) with black-box rootfinding solvers and back-propagate through the equilibrium point via implicit differentiation with constant memory.We conduct extensive experiments on four WMT machine translation benchmarks.Our main findings show that DEQNAR can indeed converge to a more accurate prediction and is a general-purpose framework that consistently helps yield substantial improvement for several strong NAR backbones.
Mind the Biases: Quantifying Cognitive Biases in Language Model Prompting

We advocate the importance of exposing uncertainty on results of language model prompting which display bias modes resembling cognitive biases, and propose to help users grasp the level of uncertainty via simple quantifying metrics.Cognitive biases in the human decision making process can lead to flawed responses when we face uncertainty.Not surprisingly, we have seen biases in language models resembling cognitive biases as a result of training on biased text, raising dangers in downstream tasks that are centered around people's lives if users trust their results too much.In this work, we reveal two bias modes leveraging cognitive biases when we prompt BERT, accompanied by two bias metrics.On a drug-drug interaction extraction task, our bias measurements reveal an error pattern similar to the availability bias when the labels for training prompts are imbalanced, and show that a toning-down transformation of the drug-drug description in a prompt can elicit a bias similar to the framing effect, warning users to distrust when prompting language models for answers. 1
Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation

Despite the huge progress in myriad generation tasks, pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts with maximization-based decoding algorithms for open-ended generation.We attribute their overestimation of token-level repetition probabilities to the learning bias: LMs capture simple repetitive patterns faster with the MLE loss.We propose self-contrastive training to penalize the output of a premature checkpoint of the same model when it incorrectly predicts repetition, which is shown to mitigate repetition effectively while maintaining fluency on two datasets.Furthermore, we find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops 1 .
Replace and Report: NLP Assisted Radiology Report Generation

Clinical practice frequently uses medical imaging for diagnosis and treatment. A significant challenge for automatic radiology report generation is that the radiology reports are long narratives consisting of multiple sentences for both abnormal and normal findings. Therefore, applying conventional image captioning approaches to generate the whole report proves to be insufficient, as these are designed to briefly describe images with short sentences. We propose a template-based approach to generate radiology reports from radiographs. Our approach involves the following: i) using a multilabel image classifier, produce the tags for the input radiograph; ii) using a transformer-based model, generate pathological descriptions (a description of abnormal findings seen on radiographs) from the tags generated in step (i); iii) using a BERT-based multi-label text classifier, find the spans in the normal report template to replace with the generated pathological descriptions; and iv) using a rule-based system, replace the identified span with the generated pathological description. We performed experiments with the two most popular radiology report datasets, IU Chest X-ray and MIMIC-CXR and demonstrated that the BLEU-1, ROUGE-L, METEOR, and CIDEr scores are better than the State-of-the-Art models by 25%, 36%, 44% and 48% respectively, on the IU X-RAY dataset. To the best of our knowledge, this is the first attempt to generate chest X-ray radiology reports by first creating small sentences for abnormal findings and then replacing them in the normal report template.
Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation

Subword segmenters like BPE operate as a preprocessing step in neural machine translation and other (conditional) language models.They are applied to datasets before training, so translation or text generation quality relies on the quality of segmentations.We propose a departure from this paradigm, called subword segmental machine translation (SSMT).SSMT unifies subword segmentation and MT in a single trainable model.It learns to segment target sentence words while jointly learning to generate target sentences.To use SSMT during inference we propose dynamic decoding, a text generation algorithm that adapts segmentations as it generates translations.Experiments across 6 translation directions show that SSMT improves chrF scores for morphologically rich agglutinative languages.Gains are strongest in the very low-resource scenario.SSMT also learns subwords that are closer to morphemes compared to baselines and proves more robust on a test set constructed for evaluating morphological compositional generalisation.
Application-Agnostic Language Modeling for On-Device ASR

On-device automatic speech recognition systems face several challenges compared to server-based systems. They have to meet stricter constraints in terms of speed, disk size and memory while maintaining the same accuracy. Often they have to serve several ap- plications with different distributions at once, such as communicating with a virtual assistant and speech-to-text. The simplest solution to serve multiple applications is to build application-specific (language) models, but this leads to an increase in memory. Therefore, we explore different data- and architecture-driven language modeling approaches to build a single application-agnostic model. We propose two novel feed-forward architectures that find an optimal trade off between different on-device constraints. In comparison to the application-specific solution, one of our novel approaches reduces the disk size by half, while maintaining speed and accuracy of the original model.
Hitachi at SemEval-2023 Task 4: Exploring Various Task Formulations Reveals the Importance of Description Texts on Human Values

This paper describes our participation in SemEval-2023 Task 4, ValueEval: Identification of Human Values behind Arguments.The aim of this task is to identify whether or not an input text supports each of the 20 pre-defined human values.Previous work on human value detection has shown the effectiveness of a sequence classification approach using BERT.However, little is known about what type of task formulation is suitable for the task.To this end, this paper explores various task formulations, including sequence classification, question answering, and question answering with chain-of-thought prompting and evaluates their performances on the shared task dataset.Experiments show that a zero-shot approach is not as effective as other methods, and there is no one approach that is optimal in every scenario.Our analysis also reveals that utilizing the descriptions of human values can help to improve performance.
Sea_and_Wine at SemEval-2023 Task 9: A Regression Model with Data Augmentation for Multilingual Intimacy Analysis

In Task 9, we are required to analyze the textual intimacy of tweets in 10 languages.We fine-tune XLM-RoBERTa (XLM-R) pretrained model to adapt to this multilingual regression task.After tentative experiments, severe class imbalance is observed in the official released dataset, which may compromise the convergence and weaken the model effect.To tackle such challenge, we take measures in two aspects.On the one hand, we implement data augmentation through machine translation to enlarge the scale of classes with fewer samples.On the other hand, we introduce focal mean square error (MSE) loss to emphasize the contributions of hard samples to total loss, thus further mitigating the impact of class imbalance on model effect.Extensive experiments demonstrate remarkable effectiveness of our strategies, and our model achieves high performance on the Pearson's correlation coefficient (CC) almost above 0.85 on validation dataset.
Learning to Rank Utterances for Query-Focused Meeting Summarization

Query-focused meeting summarization(QFMS) aims to generate a specific summary for the given query according to the meeting transcripts.Due to the conflict between long meetings and limited input size, previous works mainly adopt extract-then-summarize methods, which use extractors to simulate binary labels or ROUGE scores to extract utterances related to the query and then generate a summary.However, the previous approach fails to fully use the comparison between utterances.To the extractor, comparison orders are more important than specific scores.In this paper, we propose a Ranker-Generator framework.It learns to rank the utterances by comparing them in pairs and learning from the global orders, then uses top utterances as the generator's input.We show that learning to rank utterances helps to select utterances related to the query effectively, and the summarizer can benefit from it.Experimental results on QMSum show that the proposed model outperforms all existing multi-stage models with fewer parameters.
Pruning Pre-trained Language Models with Principled Importance and Self-regularization

Iterative pruning is one of the most effective compression methods for pre-trained language models.We discovered that finding the optimal pruning decision is an equalityconstrained 0-1 Integer Linear Programming problem.The solution to this optimization problem leads to a principled importance criterion which we use to rank parameters during iterative model pruning.To mitigate the poor generalization at high sparsity levels, we propose a self-regularization scheme where model prediction is regularized by the latest checkpoint with increasing sparsity throughout pruning.Our experiments on natural language understanding, question answering, named entity recognition, and datato-text generation with various Transformerbased PLMs show the effectiveness of the approach at various sparsity levels.