Reviewerly/exHarmony huggingface dataset:
Controlling Styles in Neural Machine Translation with Activation Prompt

Controlling styles in neural machine translation (NMT) has attracted wide attention, as it is crucial for enhancing user experience.Earlier studies on this topic typically concentrate on regulating the level of formality and achieve some progress in this area.However, they still encounter two major challenges.The first is the difficulty in style evaluation.The style comprises various aspects such as lexis, syntax, and others that provide abundant information.Nevertheless, only formality has been thoroughly investigated.The second challenge involves excessive dependence on incremental adjustments, particularly when new styles are necessary.To address both challenges, this paper presents a new benchmark and approach.A multiway stylized machine translation (MSMT) benchmark is introduced, incorporating diverse categories of styles across four linguistic domains.Then, we propose a method named style activation prompt (StyleAP) by retrieving prompts from stylized monolingual corpus, which does not require extra fine-tuning.Experiments show that StyleAP could effectively control the style of translation and achieve remarkable performance.* *Work done while Y. Wang was an intern at ByteDance.On the eleventh, an egg-sized black spot appeared on the sun.
Constrained Tuple Extraction with Interaction-Aware Network

Tuples extraction is a fundamental task for information extraction and knowledge graph construction.The extracted tuples are usually represented as knowledge triples consisting of subject, relation, and object.In practice, however, the validity of knowledge triples is associated with and changes with the spatial, temporal, or other kinds of constraints.Motivated by this observation, this paper proposes a constrained tuple extraction (CTE) task to guarantee the validity of knowledge tuples.Formally, the CTE task is to extract constrained tuples from unstructured text, which adds constraints to conventional triples.To this end, we propose an interaction-aware network.Combinatorial interactions among context-specific external features and distinct-granularity internal features are exploited to effectively mine the potential constraints.Moreover, we have built a new dataset containing totally 1,748,826 constrained tuples for training and 3656 ones for evaluation.Experiments on our dataset and the public CaRB dataset demonstrate the superiority of the proposed model.The constructed dataset and the codes are publicly available.
Pulling Out All The Full Stops: Punctuation Sensitivity in Neural Machine Translation and Evaluation

Much of the work testing machine translation systems for robustness and sensitivity has been adversarial or tended towards testing noisy input such as spelling errors, or non-standard input such as dialects.In this work, we take a step back to investigate a sensitivity problem that can seem trivial and is often overlooked: punctuation.We perform basic sentence-final insertion and deletion perturbation tests with full stops, exclamation and questions marks across source languages and demonstrate a concerning finding: commercial, production-level machine translation systems are vulnerable to mere single punctuation insertion or deletion, resulting in unreliable translations.Moreover, we demonstrate that both string-based and model-based evaluation metrics also suffer from this vulnerability, producing significantly different scores when translations only differ in a single punctuation, with model-based metrics penalizing each punctuation differently.Our work calls into question the reliability of machine translation systems and their evaluation metrics, particularly for real-world use cases, where inconsistent punctuation is often the most common and the least disruptive noise. 1
Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations

In-context learning (ICL) is an important paradigm for adapting large language models (LLMs) to new tasks, but the generalization behavior of ICL remains poorly understood.We investigate the inductive biases of ICL from the perspective of feature bias: which feature ICL is more likely to use given a set of underspecified demonstrations in which two features are equally predictive of the labels.First, we characterize the feature biases of GPT-3 models by constructing underspecified demonstrations from a range of NLP datasets and feature combinations.We find that LLMs exhibit clear feature biases-for example, demonstrating a strong bias to predict labels according to sentiment rather than shallow lexical features, like punctuation.Second, we evaluate the effect of different interventions that are designed to impose an inductive bias in favor of a particular feature, such as adding a natural language instruction or using semantically relevant label words.We find that, while many interventions can influence the learner to prefer a particular feature, it can be difficult to overcome strong prior biases.Overall, our results provide a broader picture of the types of features that ICL may be more likely to exploit and how to impose inductive biases that are better aligned with the intended task.
Teddysum at MEDIQA-Chat 2023: an analysis of fine-tuning strategy for long dialog summarization

In this paper, we introduce the design and various attempts for Task B of MEDIQA-Chat 2023.The goal of Task B in MEDIQA-Chat 2023 is to generate full clinical note from doctor-patient consultation dialogues.This task has several challenging issues, such as lack of training data, handling long dialogue inputs, and generating semi-structured clinical note which have section heads.To address these issues, we conducted various experiments and analyzed their results.We utilized the DialogLED model pre-trained on long dialogue data to handle long inputs, and we pre-trained on other dialogue datasets to address the lack of training data.We also attempted methods such as using prompts and contrastive learning for handling sections.This paper provides insights into clinical note generation through analyzing experimental methods and results, and it suggests future research directions.
ACTC: Active Threshold Calibration for Cold-Start Knowledge Graph Completion

Self-supervised knowledge-graph completion (KGC) relies on estimating a scoring model over (entity, relation, entity)-tuples, for example, by embedding an initial knowledge graph.Prediction quality can be improved by calibrating the scoring model, typically by adjusting the prediction thresholds using manually annotated examples.In this paper, we attempt for the first time cold-start calibration for KGC, where no annotated examples exist initially for calibration, and only a limited number of tuples can be selected for annotation.Our new method ACTC finds good per-relation thresholds efficiently based on a limited set of annotated tuples.Additionally to a few annotated tuples, ACTC also leverages unlabeled tuples by estimating their correctness with Logistic Regression or Gaussian Process classifiers.We also experiment with different methods for selecting candidate tuples for annotation: density-based and random selection.Experiments with five scoring models and an oracle annotator show an improvement of 7% points when using ACTC in the challenging setting with an annotation budget of only 10 tuples, and an average improvement of 4% points over different budgets.
Code-Switched Text Synthesis in Unseen Language Pairs

Existing efforts on text synthesis for codeswitching mostly require training on codeswitched texts in the target language pairs, limiting the deployment of the models to cases lacking code-switched data.In this work, we study the problem of synthesizing codeswitched texts for language pairs absent from the training data.We introduce GLOSS, a model built on top of a pre-trained multilingual machine translation model (PMMTM) with an additional code-switching module.This module, either an adapter or extra prefixes, learns code-switching patterns from codeswitched data during training, while the primary component of GLOSS, i.e., the PMMTM, is frozen.The design of only adjusting the code-switching module prevents our model from overfitting to the constrained training data for code-switching.Hence, GLOSS exhibits the ability to generalize and synthesize codeswitched texts across a broader spectrum of language pairs.Additionally, we develop a self-training algorithm on target language pairs further to enhance the reliability of GLOSS.Automatic evaluations on four language pairs show that GLOSS achieves at least 55% relative BLEU and METEOR scores improvements compared to strong baselines.Human evaluations on two language pairs further validate the success of GLOSS.* Work was done when the author interned at Amazon. 1 In this paper, we mainly focus on the sentence-level codeswitching involving only two languages.Training En-Es A superconductor levitates on a magnetic track Un superconductor levita sobre a magnetic track Inference Model There is no such thing as a perfect budget 不可能有完美的budget Es gibt kein perfect budget En-Hi Its height is only 1982 meters इसकी
Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality

Improving factual consistency of abstractive summarization has been a widely studied topic.However, most of the prior works on training factuality-aware models have ignored the negative effect it has on summary quality.We propose EFACTSUM (i.e., Effective Factual Summarization), a candidate summary generation and ranking technique to improve summary factuality without sacrificing summary quality.We show that using a contrastive learning framework with our refined candidate summaries leads to significant gains on both factuality and similarity-based metrics.Specifically, we propose a ranking strategy in which we effectively combine two metrics, thereby preventing any conflict during training.Models trained using our approach show up to 6 points of absolute improvement over the base model with respect to FactCC on XSUM and 11 points on CNN/DM, without negatively affecting either similarity-based metrics or absractiveness.
From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?

Noun compound interpretation is the task of expressing a noun compound (e.g.chocolate bunny) in a free-text paraphrase that makes the relationship between the constituent nouns explicit (e.g.bunny-shaped chocolate).We propose modifications to the data and evaluation setup of the standard task (Hendrickx et al., 2013), and show that GPT-3 solves it almost perfectly.We then investigate the task of noun compound conceptualization, i.e. paraphrasing a novel or rare noun compound.E.g., chocolate crocodile is a crocodile-shaped chocolate.This task requires creativity, commonsense, and the ability to generalize knowledge about similar concepts.While GPT-3's performance is not perfect, it is better than that of humans-likely thanks to its access to vast amounts of knowledge, and because conceptual processing is effortful for people (Connell and Lynott, 2012).Finally, we estimate the extent to which GPT-3 is reasoning about the world vs. parroting its training data.We find that the outputs from GPT-3 often have significant overlap with a large web corpus, but that the parroting strategy is less beneficial for novel noun compounds.
SUTNLP at SemEval-2023 Task 4: LG-Transformer for Human Value Detection

When we interact with other humans, human values guide us to consider the human element.As we shall see, value analysis in NLP has been applied to personality profiling but not to argument mining.As part of SemEval-2023 Shared Task 4, our system paper describes a multi-label classifier for identifying human values.Human value detection requires multilabel classification since each argument may contain multiple values.In this paper, we propose an architecture called Label Graph Transformer (LG-Transformer).LG-Transformer is a two-stage pipeline consisting of a transformer jointly encoding argument and labels and a graph module encoding and obtaining further interactions between labels.Using adversarial training, we can boost performance even further.Our best method scored 50.00 % using F1 score on the test set, which is 7.8 % higher than the best baseline method.Our code is publicly available on Github. 1
Weakly supervised hierarchical multi-task classification of customer questions

Jitenkumar Rana, Promod Yenigalla, Chetan Aggarwal, Sandeep Sricharan Mukku, Manan Soni, Rashmi Patange. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track). 2023.
Pronunciation Ambiguities in Japanese Kanji

Japanese writing is a complex system, and a large part of the complexity resides in the use of kanji.A single kanji character in modern Japanese may have multiple pronunciations, either as native vocabulary or as words borrowed from Chinese.This causes a problem for text-tospeech synthesis (TTS) because the system has to predict which pronunciation of each kanji character is appropriate in the context.The problem is called homograph disambiguation.To solve the problem, this research provides a new annotated Japanese single kanji character pronunciation data set and describes an experiment using the logistic regression (LR) classifier.A baseline is computed to compare with the LR classifier accuracy.This experiment provides the first experimental research in Japanese single kanji homograph disambiguation.The annotated Japanese data is freely released to the public to support further work. IntroductionJapanese uses a mixed writing system with three distinct scripts and one romanization.Kanji 漢字 is the writing script that borrows directly from Chinese characters which were introduced in Japan from China through Korea from the third century CE.There are 2,136 commonly used kanji characters termed Joyo kanji in present-day Japanese. 1 A single kanji character in modern Japanese may have multiple pronunciations derived from the linguistic history of the kanji characters as either native vocabulary words or as terms borrowed from Chinese.For instance, the 1 https://kanji.jitenon.jp/cat/joyo.htmlkanji character 山 'mountain' can be read as either the native Japanese word yama or the Chinesederived term san.The native Japanese pronunciations of the kanji character 文 'letter, sentence, writings' are humi, aya, and kaza, while Chinese borrowed pronunciations are bun and mon.Because a kanji character has multiple pronunciations, to predict the appropriate pronunciation for each kanji character, a text-tospeech synthesis engine must select the appropriate reading.This is a form of homograph disambiguation.This research is a computational study of Japanese kanji homograph disambiguation.Recent research in homograph disambiguation in Japanese is limited because of the lack of extensive data sets that include comprehensive pronunciations for the most commonly used kanji characters.The goal of this research is to fill this void, make new data sets to conduct the analysis of kanji characters with multiple pronunciations, and use the computational methodology to test the data set to lay a foundation for computational research on Japanese kanji homographs in the future. Japanese writing scriptsThe Japanese writing system uses three different scripts, Chinese characters (kanji), and two kana systems: hiragana and katakana, which are derivatives of Chinese characters.Hiragana resulted from the cursive style of writing Chinese characters, while katakana developed from the abbreviation of Chinese characters.Roughly speaking, kanji are used for content words such as nouns, stems of adjectives，and verbs, whereas
Decomposed scoring of CCG dependencies

In statistical parsing with ccg, the standard evaluation method is based on predicate-argument structure and evaluates dependencies labelled in part by lexical categories.When a predicate has multiple argument slots that can be filled, the same lexical category is used for the label of multiple dependencies.In this paper, we show that this evaluation can result in disproportionate penalization of supertagging errors and obfuscate the truly erroneous dependencies.Enabled by the compositional nature of ccg lexical categories, we propose decomposed scoring based on subcategorial labels to address this.To evaluate our scoring method, we engage fellow categorial grammar researchers in two English-language judgement tasks: (1) directly ranking the outputs of the standard and experimental scoring methods; and (2) determining which of two sentences has the better parse in cases where the two scoring methods disagree on their ranks.Overall, the judges prefer decomposed scoring in each task; but there is substantial disagreement among the judges in 24% of the given cases, pointing to potential issues with parser evaluations in general.
DS at SemEval-2023 Task 10: Explaining Online Sexism using Transformer based Approach

In this paper, I describe the approach used in the SemEval 2023 -Task 10 Explainable Detection of Online Sexism (EDOS) competition (Kirk et al., 2023).I use different transformer models, including BERT and RoBERTa which were fine-tuned on the EDOS dataset to classify text into different categories of sexism.I participated in three subtasks: subtask A is to classify given text as either sexist or not, while subtask B is to identify the specific category of sexism, such as (1) threats, (2) derogation, (3) animosity, (4) prejudiced discussions.Finally, subtask C involves predicting a finegrained vector representation of sexism, which included information about the severity, target and type of sexism present in the text.The use of transformer models allows the system to learn from the input data and make predictions on unseen text.By fine-tuning the models on the EDOS dataset, the system can improve its performance on the specific task of detecting online sexism.I got the following macro F1 scores: subtask A:77.
Cross Encoding as Augmentation: Towards Effective Educational Text Classification

Text classification in education, usually called auto-tagging, is the automated process of assigning relevant tags to educational content, such as questions and textbooks.However, auto-tagging suffers from a data scarcity problem, which stems from two major challenges: 1) it possesses a large tag space and 2) it is multi-label.Though a retrieval approach is reportedly good at low-resource scenarios, there have been fewer efforts to directly address the data scarcity problem.To mitigate these issues, here we propose a novel retrieval approach CEAA that provides effective learning in educational text classification.Our main contributions are as follows: 1) we leverage transfer learning from question-answering datasets, and 2) we propose a simple but effective data augmentation method introducing cross-encoder style texts to a bi-encoder architecture for more efficient inference.An extensive set of experiments shows that our proposed method is effective in multi-label scenarios and low-resource tags compared to state-of-the-art models.
Exploiting Hierarchically Structured Categories in Fine-grained Chinese Named Entity Recognition

Chinese Named Entity Recognition (CNER) is a widely used technology in various applications.While recent studies have focused on utilizing additional information of the Chinese language and characters to enhance CNER performance, this paper focuses on a specific aspect of CNER known as fine-grained CNER (FG-CNER).FG-CNER involves the use of hierarchical, fine-grained categories (e.g.Person-MovieStar) to label named entities.To promote research in this area, we introduce the FiNE dataset, a dataset for FG-CNER consisting of 30,000 sentences from various domains and containing 67,651 entities in 54 fine-grained flattened hierarchical categories.Additionally, we propose SoftFiNE, a novel approach for FG-CNER that utilizes a custom-designed relevance scoring function based on label structures to learn the potential relevance between different flattened hierarchical labels.Our experimental results demonstrate that the proposed SoftFiNE method outperforms the state-of-theart baselines on the FiNE dataset.Furthermore, we conduct extensive experiments on three other datasets, including OntoNotes 4.0, Weibo, and Resume, where SoftFiNE achieved stateof-the-art performance on all three datasets.
Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring

Automated essay scoring (AES) aims to score essays written for a given prompt, which defines the writing topic.Most existing AES systems assume to grade essays of the same prompt as used in training and assign only a holistic score.However, such settings conflict with real-education situations; pre-graded essays for a particular prompt are lacking, and detailed trait scores of sub-rubrics are required.Thus, predicting various trait scores of unseenprompt essays (called cross-prompt essay trait scoring) is a remaining challenge of AES.In this paper, we propose a robust model: promptand trait relation-aware cross-prompt essay trait scorer.We encode prompt-aware essay representation by essay-prompt attention and utilizing the topic-coherence feature extracted by the topic-modeling mechanism without access to labeled data; therefore, our model considers the prompt adherence of an essay, even in a crossprompt setting.To facilitate multi-trait scoring, we design trait-similarity loss that encapsulates the correlations of traits.Experiments prove the efficacy of our model, showing state-of-theart results for all prompts and traits.Significant improvements in low-resource-prompt and inferior traits further indicate our model's strength.
TeCS: A Dataset and Benchmark for Tense Consistency of Machine Translation

Tense inconsistency frequently occurs in machine translation.However, there are few criteria to assess the model's mastery of tense prediction from a linguistic perspective.In this paper, we present a parallel tense test set, containing French-English 552 utterances 1 .We also introduce a corresponding benchmark, tense prediction accuracy.With the tense test set and the benchmark, researchers are able to measure the tense consistency performance of machine translation systems for the first time.
Multilingual Multifaceted Understanding of Online News in Terms of Genre, Framing, and Persuasion Techniques

We present a new multilingual multifacet dataset of news articles, each annotated for genre (objective news reporting vs. opinion vs. satire), framing (what key aspects are highlighted), and persuasion techniques (logical fallacies, emotional appeals, ad hominem attacks, etc.).The persuasion techniques are annotated at the span level, using a taxonomy of 23 fine-grained techniques grouped into 6 coarse categories.The dataset contains 1,612 news articles covering recent news on current topics of public interest in six European languages (English, French, German, Italian, Polish, and Russian), with more than 37k annotated spans of persuasion techniques.We describe the dataset and the annotation process, and we report the evaluation results of multilabel classification experiments using stateof-the-art multilingual transformers at different levels of granularity: token-level, sentencelevel, paragraph-level, and document-level.