Reviewerly/exHarmony huggingface dataset:
CSECU-DSG at SemEval-2023 Task 4: Fine-tuning DeBERTa Transformer Model with Cross-fold Training and Multi-sample Dropout for Human Values Identification

Human values identification from a set of argument is becoming a prominent area of research in argument mining.Among some options, values convey what may be the most desirable and widely accepted answer.The diversity of human beliefs, random texture, and implicit meaning within the arguments make it more difficult to identify human values from the arguments.To address these challenges, SemEval-2023 Task 4 introduced a shared task ValueEval focusing on identifying human value categories based on given arguments.This paper presents our participation in this task where we propose a fine-tuned DeBERTa transformers-based classification approach to identify the desired human value category.We utilize different training strategies with the fine-tuned DeBERTa model to enhance contextual representation on this downstream task.Our proposed method achieved competitive performance among the participants' methods.
Detection and Mitigation of the Negative Impact of Dataset Extractivity on Abstractive Summarization

In text summarization, extractivity is defined as a measurement of the degree of overlap between a source document and its summary.Previous research has shown that the extractivity level of training data can influence both output extractivity and the amount of factual information (i.e.faithfulness) in outputs for abstractive summarization.However, it remains unclear if and how extractivity impacts the performance of abstractive models.In this work, we investigate the relationship between dataset extractivity and model performance by comparing the performance of trained models under different degrees of extractivity.We find that while low levels of extractivity can improve performance, as extractivity increases, performance is negatively impacted.Furthermore, through an analysis of the model's copy continuity of content, we discover that higher extractivity leads to a greater tendency for the model to copy text continuously from the source document rather than identifying and summarizing important content that should be covered in the target summary.To address these issues, we propose a simple and effective method to design copy labels for fixing the model's copying behaviors and train the model with a copy mechanism.The experimental results illustrate the effectiveness of our strategy in alleviating the negative impact on model performance resulting from high dataset extractivity, and that our method outperforms several competitive baselines.
Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information

The long-standing one-to-many issue of the open-domain dialogues poses significant challenges for automatic evaluation methods, i.e., there may be multiple suitable responses which differ in semantics for a given conversational context.To tackle this challenge, we propose a novel learning-based automatic evaluation metric (CMN), which can robustly evaluate open-domain dialogues by augmenting Conditional Variational Autoencoders (CVAEs) with a Next Sentence Prediction (NSP) objective and employing Mutual Information (MI) to model the semantic similarity of text in the latent space.Experimental results on two opendomain dialogue datasets demonstrate the superiority of our method compared with a wide range of baselines, especially in handling responses which are distant to the golden reference responses in semantics.
Bhasa-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages

We create publicly available language identification (LID) datasets and models in all 22 Indian languages listed in the Indian constitution in both native-script and romanized text.First, we create Bhasha-Abhijnaanam, a language identification test set for native-script as well as romanized text which spans all 22 Indic languages.We also train IndicLID, a language identifier for all the above-mentioned languages in both native and romanized script.For native-script text, it has better language coverage than existing LIDs and is competitive or better than other LIDs.IndicLID is the first LID for romanized text in Indian languages.Two major challenges for romanized text LID are the lack of training data and low-LID performance when languages are similar.We provide simple and effective solutions to these problems.In general, there has been limited work on romanized text in any language, and our findings are relevant to other languages that need romanized language identification.Our models are publicly available at https:
BERTastic at SemEval-2023 Task 3: Fine-Tuning Pretrained Multilingual Transformers Does Order Matter?

The naïve approach for fine-tuning pretrained deep learning models on downstream tasks involves feeding them mini-batches of randomly sampled data.In this paper, we propose a more elaborate method for fine-tuning Pretrained Multilingual Transformers (PMTs) on multilingual data.Inspired by the success of curriculum learning approaches, we investigate the significance of fine-tuning PMTs on multilingual data in a sequential fashion language by language.Unlike the curriculum learning paradigm where the model is presented with increasingly complex examples, we do not adopt a notion of "easy" and "hard" samples.Instead, our experiments draw insight from psychological findings on how the human brain processes new information and the persistence of newly learned concepts.We perform our experiments on a challenging news-framing dataset that contains texts in six languages.Our proposed method outperforms the naïve approach by achieving improvements of 2.57% in terms of F1 score.Even when we supplement the naïve approach with recency fine-tuning, we still achieve an improvement of 1.34% with a 3.63% convergence speed-up.Moreover, we are the first to observe an interesting pattern in which deep learning models exhibit a human-like primacyrecency effect.
Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities

Although we have witnessed impressive progress in Semantic Role Labeling (SRL), most of the research in the area is carried out assuming that the majority of predicates are verbs.Conversely, predicates can also be expressed using other parts of speech, e.g., nouns and adjectives.However, non-verbal predicates appear in the benchmarks we commonly use to measure progress in SRL less frequently than in some real-world settings -newspaper headlines, dialogues, and tweets, among others.In this paper, we put forward a new PropBank dataset which boasts wide coverage of multiple predicate types.Thanks to it, we demonstrate empirically that standard benchmarks do not provide an accurate picture of the current situation in SRL and that state-of-the-art systems are still incapable of transferring knowledge across different predicate types.Having observed these issues, we also present a novel, manually-annotated challenge set designed to give equal importance to verbal, nominal, and adjectival predicate-argument structures.We use such dataset to investigate whether we can leverage different linguistic resources to promote knowledge transfer.In conclusion, we claim that SRL is far from "solved", and its integration with other semantic tasks might enable significant improvements in the future, especially for the long tail of non-verbal predicates, thereby facilitating further research on SRL for non-verbal predicates.We release our software and datasets at
Ginn-Khamov at SemEval-2023 Task 6, Subtask B: Legal Named Entities Extraction for Heterogenous Documents

This paper describes our submission to SemEval-2023 Task 6, Subtask B, a shared task on performing Named Entity Recognition in legal documents for specific legal entity types.Documents are divided into the preamble and judgement texts, and certain entity types should only be tagged in one of the two text sections.To address this challenge, our team proposes a token classification model that is augmented with information about the document type, which achieves greater performance than the non-augmented system.
Claim-Dissector: An Interpretable Fact-Checking System with Joint Re-ranking and Veracity Prediction

We present Claim-Dissector: a novel latent variable model for fact-checking and analysis, which given a claim and a set of retrieved evidences jointly learns to identify: (i) the relevant evidences to the given claim, (ii) the veracity of the claim.We propose to disentangle the per-evidence relevance probability and its contribution to the final veracity probability in an interpretable way -the final veracity probability is proportional to a linear ensemble of per-evidence relevance probabilities.In this way, the individual contributions of evidences towards the final predicted probability can be identified.In per-evidence relevance probability, our model can further distinguish whether each relevant evidence is supporting (S) or refuting (R) the claim.This allows to quantify how much the S/R probability contributes to the final verdict or to detect disagreeing evidence.Despite its interpretable nature, our system achieves results competitive with state-of-theart on the FEVER dataset, as compared to typical two-stage system pipelines, while using significantly fewer parameters.It also sets new state-of-the-art on FAVIQ and RealFC datasets.Furthermore, our analysis shows that our model can learn fine-grained relevance cues while using coarse-grained supervision, and we demonstrate it in 2 ways.(i) We show that our model can achieve competitive sentence recall while using only paragraph-level relevance supervision.(ii) Traversing towards the finest granularity of relevance, we show that our model is capable of identifying relevance at the token level.To do this, we present a new benchmark TLR-FEVER focusing on token-level interpretability -humans annotate tokens in relevant evidences they considered essential when making their judgment.Then we measure how similar are these annotations to the tokens our model is focusing on. 1 1 https://github.com/KNOT-FIT-BUT/ClaimDissector.
Context-Aware Document Simplification

To date, most work on text simplification has focused on sentence-level inputs.Early attempts at document simplification merely applied these approaches iteratively over the sentences of a document.However, this fails to coherently preserve the discourse structure, leading to suboptimal output quality.Recently, strategies from controllable simplification have been leveraged to achieve state-ofthe-art results on document simplification by first generating a document-level plan (a sequence of sentence-level simplification operations) and using this plan to guide sentencelevel simplification downstream.However, this is still limited in that the simplification model has no direct access to the local intersentence document context, likely having a negative impact on surface realisation.We explore various systems that use document context within the simplification process itself, either by iterating over larger text units or by extending the system architecture to attend over a high-level representation of document context.In doing so, we achieve state-of-the-art performance on the document simplification task, even when not relying on plan-guidance.Further, we investigate the performance and efficiency tradeoffs of system variants and make suggestions of when each should be preferred.
JHU IWSLT 2023 Dialect Speech Translation System Description

This paper presents JHU's submissions to the IWSLT 2023 dialectal and low-resource track of Tunisian Arabic to English speech translation.The Tunisian dialect lacks formal orthography and abundant training data, making it challenging to develop effective speech translation (ST) systems.To address these challenges, we explore the integration of large pretrained machine translation (MT) models, such as mBART and NLLB-200 in both end-to-end (E2E) and cascaded speech translation (ST) systems.We also improve the performance of automatic speech recognition (ASR) through the use of pseudo-labeling data augmentation and channel matching on telephone data.Finally, we combine our E2E and cascaded ST systems with Minimum Bayes-Risk decoding.Our combined system achieves a BLEU score of 21.6 and 19.1 on test2 and test3, respectively.
UTB-NLP at SemEval-2023 Task 3: Weirdness, Lexical Features for Detecting Categorical Framings, and Persuasion in Online News

Nowadays, persuasive messages are increasingly frequent in social networks, which generates particular concern in several communities, given that persuasion seeks to guide others towards adopting ideas, attitudes, or actions that they consider to be beneficial to themselves.The efficient detection of news genre categories, detection of framing, and detection of persuasion techniques require several scientific disciplines, such as computational linguistics and sociology.Here we illustrate how we use lexical features given a news article to determine whether it is an opinion piece, aims to report factual news, or is satire.This paper presents a novel strategy for communication based on Lexical Weirdness.The results are part of our participation in Sub-Tasks 1 and 2 in SemEval 2023 Task 3.
Contrastive Learning with Adversarial Examples for Alleviating Pathology of Language Model

Neural language models have achieved superior performance.However, these models also suffer from the pathology of overconfidence in the out-of-distribution examples, potentially making the model difficult to interpret and making the interpretation methods fail to provide faithful attributions.In this paper, we explain the model pathology from the view of sentence representation and argue that the counter-intuitive bias degree and direction of the out-of-distribution examples' representation cause the pathology.We propose a Contrastive learning regularization method using Adversarial examples for Alleviating the Pathology (ConAAP), which calibrates the sentence representation of out-of-distribution examples.ConAAP generates positive and negative examples following the attribution results and utilizes adversarial examples to introduce direction information in regularization.Experiments show that ConAAP effectively alleviates the model pathology while slightly impacting the generalization ability on in-distribution examples and thus helps interpretation methods obtain more faithful results.
Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation

Multimodal machine translation (MMT) aims to improve translation quality by incorporating information from other modalities, such as vision. Previous MMT systems focus on better access and use of visual information and tend to validate their methods on image-related datasets. However, these studies face two challenges. First, they can only utilize a limited amount of data that is composed of bilingual texts and images (referred to as “triple data”), which is scarce. Second, current benchmarks for MMT are restricted and do not correspond to realistic scenarios. Therefore, this paper correspondingly establishes new methods and a new dataset for MMT. We propose a novel framework for MMT that addresses these challenges by utilizing large-scale non-triple data, such as monolingual image-text and parallel text-only data. Additionally, we construct a new e-commercial multimodal translation dataset, named EMMT, of which the test set is specifically designed to include ambiguous words that require visual context for accurate translation. Experiments show that our method is well-suited for real-world scenarios and can significantly improve translation performance with more non-triple data. In addition, our model also rivals or surpasses various SOTA models in conventional multimodal translation benchmarks.
Evaluating pragmatic abilities of image captioners on A3DS

Evaluating grounded neural language model performance with respect to pragmatic qualities like the trade off between truthfulness, contrastivity and overinformativity of generated utterances remains a challenge in absence of data collected from humans.To enable such evaluation, we present a novel open source image-text dataset "Annotated 3D Shapes" (A3DS) comprising over nine million exhaustive natural language annotations and over 12 million variablegranularity captions for the 480,000 images provided by Burgess and Kim (2018).We showcase the evaluation of pragmatic abilities developed by a task-neutral image captioner fine-tuned in a multi-agent communication setting to produce contrastive captions.The evaluation is enabled by the dataset because the exhaustive annotations allow to quantify the presence of contrastive features in the model's generations.We show that the model develops human-like patterns (informativity, brevity, over-informativity for specific features (e.g., shape, color biases)).
Rapid Diffusion: Building Domain-Specific Text-to-Image Synthesizers with Fast Inference Speed

Text-to-Image Synthesis (TIS) aims to generate images based on textual inputs.Recently, several large pre-trained diffusion models have been released to create high-quality images with pre-trained text encoders and diffusionbased image synthesizers.However, popular diffusion-based models from the open-source community cannot support industrial domainspecific applications due to the lack of entity knowledge and low inference speed.In this paper, we propose Rapid Diffusion, a novel framework for training and deploying superresolution, text-to-image latent diffusion models with rich entity knowledge injected and optimized networks.Furthermore, we employ BladeDISC, an end-to-end Artificial Intelligence (AI) compiler, and FlashAttention techniques to optimize computational graphs of the generated models for online deployment.Experiments verify the effectiveness of our approach in terms of image quality and inference speed.In addition, we present industrial use cases and integrate Rapid Diffusion to an AI platform to show its practical values.
CoAug: Combining Augmentation of Labels and Labelling Rules

Collecting labeled data for Named Entity Recognition (NER) tasks is challenging due to the high cost of manual annotations.Instead, researchers have proposed few-shot self-training and rule-augmentation techniques to minimize the reliance on large datasets.However, inductive biases and restricted logical language lexicon, respectively, can limit the ability of these models to perform well.In this work, we propose CoAug, a co-augmentation framework that allows us to improve few-shot models and ruleaugmentation models by bootstrapping predictions from each model.By leveraging rules and neural model predictions to train our models, we complement the benefits of each and achieve the best of both worlds.In our experiments, we show that our best CoAug model can outperform strong weak-supervision-based NER models at least by 6.5 F1 points on the BC5CDR, NCBI-Disease, WikiGold, and CoNLL-2003 datasets. 1
Text Adversarial Purification as Defense against Adversarial Attacks

Adversarial purification is a successful defense mechanism against adversarial attacks without requiring knowledge of the form of the incoming attack.Generally, adversarial purification aims to remove the adversarial perturbations therefore can make correct predictions based on the recovered clean samples.Despite the success of adversarial purification in the computer vision field that incorporates generative models such as energy-based models and diffusion models, using purification as a defense strategy against textual adversarial attacks is rarely explored.In this work, we introduce a novel adversarial purification method that focuses on defending against textual adversarial attacks.With the help of language models, we can inject noise by masking input texts and reconstructing the masked texts based on the masked language models.In this way, we construct an adversarial purification process for textual models against the most widely used word-substitution adversarial attacks.We test our proposed adversarial purification method on several strong adversarial attack methods including Textfooler and BERT-Attack and experimental results indicate that the purification algorithm can successfully defend against strong word-substitution attacks.
Fine-grained Text Style Transfer with Diffusion-Based Language Models

Diffusion probabilistic models have shown great success in generating high-quality images controllably, and researchers have tried to utilize this controllability into text generation domain. Previous works on diffusion-based language models have shown that they can be trained without external knowledge (such as pre-trained weights) and still achieve stable performance and controllability. In this paper, we trained a diffusion-based model on StylePTB dataset, the standard benchmark for fine-grained text style transfers. The tasks in StylePTB requires much more refined control over the output text compared to tasks evaluated in previous works, and our model was able to achieve state-of-the-art performance on StylePTB on both individual and compositional transfers. Moreover, our model, trained on limited data from StylePTB without external knowledge, outperforms previous works that utilized pretrained weights, embeddings, and external grammar parsers, and this may indicate that diffusion-based language models have great potential under low-resource settings.
Evaluating Factuality in Cross-lingual Summarization

Cross-lingual summarization aims to help people efficiently grasp the core idea of the document written in a foreign language.Modern text summarization models generate highly fluent but often factually inconsistent outputs, which has received heightened attention in recent research.However, the factual consistency of cross-lingual summarization has not been investigated yet.In this paper, we propose a crosslingual factuality dataset by collecting human annotations of reference summaries as well as generated summaries from models at both summary level and sentence level.Furthermore, we perform the fine-grained analysis and observe that over 50% of generated summaries and over 27% of reference summaries contain factual errors with characteristics different from monolingual summarization.Existing evaluation metrics for monolingual summarization require translation to evaluate the factuality of crosslingual summarization and perform differently at different tasks and levels.Finally, we adapt the monolingual factuality metrics as an initial step towards the automatic evaluation of summarization factuality in cross-lingual settings.Our dataset and code are available at https: //github.com/kite99520/Fact_CLS. * Equal contribution.by the General Administration of Customs on the 10th show that the total value of China's imports and exports in February was 260.43 billion U.S. dollars, up by 29.4%.Among them, the export was U.S. dollars 114.47 billion, up by 18.4%; Imports reached 145.96 billion U.S. dollars, up by 39.6 %.The trade deficit of 31.49 billion U.S. dollars was the largest in nearly a decade.) Summaries TNCLS: China's exports exceeded 100 billion US dollars in February, the biggest trade deficit in nearly 10 years.CLSMS: China's trade deficit in the past 10 years is the largest in nearly 10 years.CLSMT: In February, the total import and export value of China's foreign trade increased by 29.4 % compared with the same period last year.ATS: China's foreign trade deficit in February was 26.4 billion US dollars, the biggest in 10 years.