Reviewerly/exHarmony huggingface dataset:
Improving HowNet-Based Chinese Word Sense Disambiguation with Translations

Word sense disambiguation (WSD) is the task of identifying the intended sense of a word in context.While prior work on unsupervised WSD has leveraged lexical knowledge bases, such as WordNet and BabelNet, these resources have proven to be less effective for Chinese.Instead, the most widely used lexical knowledge base for Chinese is HowNet.Previous HowNetbased WSD methods have not exploited contextual translation information.In this paper, we present the first HowNet-based WSD system which combines monolingual contextual information from a pretrained neural language model with bilingual information obtained via machine translation and sense translation information from HowNet.The results of our evaluation experiment on a test set from prior work demonstrate that our new method achieves a new state of the art for unsupervised Chinese WSD.
KE-GCL: Knowledge Enhanced Graph Contrastive Learning for Commonsense Question Answering

Commonsense question answering (CQA) aims to choose the correct answers for commonsense questions.Most existing works focus on extracting and reasoning over external knowledge graphs (KG).However, the noise in KG prevents these models from learning effective representations.In this paper, we propose a Knowledge Enhanced Graph Contrastive Learning model (KE-GCL) by incorporating the contextual descriptions of entities and adopting a graph contrastive learning scheme.Specifically, for QA pairs we represent the knowledge from KG and contextual descriptions.Then, the representations of contextual descriptions as context nodes are inserted into KG, forming the knowledge-enhanced graphs.Moreover, we design a contrastive learning method on graphs.For knowledge-enhanced graphs, we build their augmented views with an adaptive sampling strategy.After that, we reason over graphs to update their representations by scattering edges and aggregating nodes.To further improve GCL, hard graph negatives are chosen based on incorrect answers.Extensive experiments on two benchmark datasets demonstrate the effectiveness of our proposed KE-GCL, which outperforms previous methods consistently 1 .
On the Role of Bidirectionality in Language Model Pre-Training

Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult. In this work, we focus on bidirectionality as a key factor that differentiates existing approaches, and present a comprehensive study of its role in next token prediction, text infilling, zero-shot priming and fine-tuning. We propose a new framework that generalizes prior approaches, including fully unidirectional models like GPT, fully bidirectional models like BERT, and hybrid models like CM3 and prefix LM. Our framework distinguishes between two notions of bidirectionality (bidirectional context and bidirectional attention) and allows us to control each of them separately. We find that the optimal configuration is largely application-dependent (e.g., bidirectional attention is beneficial for fine-tuning and infilling, but harmful for next token prediction and zero-shot priming). We train models with up to 6.7B parameters, and find differences to remain consistent at scale. While prior work on scaling has focused on left-to-right autoregressive models, our results suggest that this approach comes with some trade-offs, and it might be worthwhile to develop very large bidirectional models.
KOLD: Korean Offensive Language Dataset

Warning: this paper contains content that may be offensive or upsetting.Recent directions for offensive language detection are hierarchical modeling, identifying the type and the target of offensive language, and interpretability with offensive span annotation and prediction.These improvements are focused on English and do not transfer well to other languages because of cultural and linguistic differences.In this paper, we present the Korean Offensive Language Dataset (KOLD) comprising 40,429 comments, which are annotated hierarchically with the type and the target of offensive language, accompanied by annotations of the corresponding text spans.We collect the comments from NAVER news and YouTube platform and provide the titles of the articles and videos as the context information for the annotation process.We use these annotated comments as training data for Korean BERT and RoBERTa models and find that they are effective at offensiveness detection, target classification, and target span detection while having room for improvement for target group classification and offensive span detection.We discover that the target group distribution differs drastically from the existing English datasets, and observe that providing the context information improves the model performance in offensiveness detection (+0.3), target classification (+1.5), and target group classification (+13.1).We publicly release the dataset and baseline models.
English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings

Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space.Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel sentences.In this work, we propose mSimCSE, which extends SimCSE (Gao et al., 2021) to multilingual settings and reveal that contrastive learning on English data can surprisingly learn high-quality universal cross-lingual sentence embeddings without any parallel data.In unsupervised and weakly supervised settings, mSim-CSE significantly improves previous sentence embedding methods on cross-lingual retrieval and multilingual STS tasks.The performance of unsupervised mSimCSE is comparable to fully supervised methods in retrieving lowresource languages and multilingual STS.The performance can be further enhanced when cross-lingual NLI data is available. 1
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings

Contrastive learning has been extensively studied in sentence embedding learning, which assumes that the embeddings of different views of the same sentence are closer.The constraint brought by this assumption is weak, and a good sentence representation should also be able to reconstruct the original sentence fragments.Therefore, this paper proposes an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings, termed InfoCSE.InfoCSE forces the representation of [CLS] positions to aggregate denser sentence information by introducing an additional Masked language model task and a well-designed network.We evaluate the proposed InfoCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task.Experimental results show that InfoCSE outperforms SimCSE by an average Spearman correlation of 2.60% on BERT-base, and 1.77% on BERT-large, achieving state-of-the-art results among unsupervised sentence representation learning methods.Our code are available at github.com/caskcsg/sentemb/tree/main/InfoCSE.
Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature

Literary translation is a culturally significant task, but it is bottlenecked by the small number of qualified literary translators relative to the many untranslated works published around the world.Machine translation (MT) holds potential to complement the work of human translators by improving both training procedures and their overall efficiency.Literary translation is less constrained than more traditional MT settings since translators must balance meaning equivalence, readability, and critical interpretability in the target language.This property, along with the complex discourse-level context present in literary texts, also makes literary MT more challenging to computationally model and evaluate.To explore this task, we collect a dataset (PAR3) of non-English language novels in the public domain, each aligned at the paragraph level to both human and automatic English translations.Using PAR3, we discover that expert literary translators prefer reference human translations over machinetranslated paragraphs at a rate of 84%, while state-of-the-art automatic MT metrics do not correlate with those preferences.The experts note that MT outputs contain not only mistranslations, but also discourse-disrupting errors and stylistic inconsistencies.To address these problems, we train a post-editing model whose output is preferred over normal MT output at a rate of 69% by experts.We publicly release PAR3 to spur future research into literary MT. 1
SEAL: Interactive Tool for Systematic Error Analysis and Labeling

With the advent of Transformers, large language models (LLMs) have saturated wellknown NLP benchmarks and leaderboards with high aggregate performance.However, many times these models systematically fail on tail data or rare groups not obvious in aggregate evaluation.Identifying such problematic data groups is even more challenging when there are no explicit labels (e.g., ethnicity, gender, etc.) and further compounded for NLP datasets due to the lack of visual features to characterize failure modes (e.g., Asian males, animals indoors, waterbirds on land etc.).This paper introduces an interactive Systematic Error Analysis and Labeling (SEAL) tool that uses a twostep approach to first identify high error slices of data and then in the second step introduce methods to give human-understandable semantics to those under-performing slices.We explore a variety of methods for coming up with coherent semantics for the error groups using language models for semantic labeling and a text-to-image model for generating visual features.SEAL toolkit and demo screencast is available at https://huggingface.co/spaces/ nazneen/seal.
Retrieval Augmented Visual Question Answering with Outside Knowledge

Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images.Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance.Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion.Our experiments show that our scheme outperforms recent OK-VQA systems with strong DPR for retrieval.We also introduce new diagnostic metrics to analyze how retrieval and generation interact.The strong retrieval ability of our model significantly reduces the number of retrieved documents needed in training, yielding significant benefits in answer quality and computation required for training.
Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages

Scaling multilingual representation learning beyond the hundred most frequent languages is challenging, in particular to cover the long tail of low-resource languages. We move away from the popular one-for-all multilingual models and focus on training multiple language (family) specific representations, but most prominently enable all languages to still be encoded in the same representational space. We focus on teacher-student training, allowing all encoders to be mutually compatible for bitext mining, and enabling fast learning of new languages. We also combine supervised and self-supervised training, allowing encoders to take advantage of monolingual training data.Our approach significantly outperforms the original LASER encoder. We study very low-resource languages and handle 44 African languages, many of which are not covered by any other model. For these languages, we train sentence encoders and mine bitexts. Adding these mined bitexts yielded an improvement of 3.8 BLEU for NMT into English.
Adaptive Graph Convolutional Network for Knowledge Graph Entity Alignment

Entity alignment (EA) aims to identify equivalent entities from different Knowledge Graphs (KGs), which is a fundamental task for integrating KGs.Throughout its development, Graph Convolutional Network (GCN) has become one of the mainstream methods for EA.The key idea that GCN works in EA is that entities with similar neighbor structures are highly likely to be aligned.However, the noisy neighbors of entities transfer invalid information, drown out equivalent information, lead to inaccurate entity embeddings, and finally reduce the performance of EA.In this paper, we propose a lightweight framework with no training parameters for both supervised and unsupervised EA.Based on the Sinkhorn algorithm, we design a reliability measure for pseudo equivalent entities and propose Adaptive Graph Convolutional Network to deal with neighbor noises in GCN.During the training, the network dynamically updates the adaptive weights of relation triples to weaken the propagation of noises.Extensive experiments on benchmark datasets demonstrate that our framework outperforms the state-of-the-art methods in both supervised and unsupervised settings.
Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field

Ultra-fine entity typing (UFET) aims to predict a wide range of type phrases that correctly describe the categories of a given entity mention in a sentence.Most recent works infer each entity type independently, ignoring the correlations between types, e.g., when an entity is inferred as a president, it should also be a politician and a leader.To this end, we use an undirected graphical model called pairwise conditional random field (PCRF) to formulate the UFET problem, in which the type variables are not only unarily influenced by the input but also pairwisely relate to all the other type variables.We use various modern backbones for entity typing to compute unary potentials, and derive pairwise potentials from type phrase representations that both capture prior semantic information and facilitate accelerated inference.We use mean-field variational inference for efficient type inference on very large type sets and unfold it as a neural network module to enable end-to-end training.Experiments on UFET show that the Neural-PCRF consistently outperforms its backbones with little cost and results in a competitive performance against crossencoder based SOTA while being thousands of times faster.We also find Neural-PCRF effective on a widely used fine-grained entity typing dataset with a smaller type set.We pack Neural-PCRF as a network module that can be plugged onto multi-label type classifiers with ease and release it in github.com/modelscope/adaseq/examples/NPCRF.
SMARTAVE: Structured Multimodal Transformer for Product Attribute Value Extraction

Automatic product attribute value extraction refers to the task of identifying values of an attribute from the product information.Product attributes are essential in improving online shopping experience for customers.Most existing methods focus on extracting attribute values from product title and description.However, in many real-world applications, a product is usually represented by multiple modalities beyond title and description, such as product specifications, text and visual information from the product image, etc.In this paper, we propose SMARTAVE, a Structure Mltimodal trAnsformeR for producT Attribute Value Extraction, which jointly encodes the structured product information from multiple modalities.Specifically, in SMARTAVE encoder, we introduce hyper-tokens to represent the modality-level information, and localtokens to represent the original text and visual inputs.Structured attention patterns are designed among the hyper-tokens and localtokens for learning effective product representation.The attribute values are then extracted based on the learned embeddings.We conduct extensive experiments on two multimodal product datasets.Experimental results demonstrate the superior performance of the proposed approach over several state-of-the-art methods.Ablation studies validate the effectiveness of the structured attentions in modeling the multimodal product information.
IndicXNLI: Evaluating Multilingual Inference for Indian Languages

While Indic NLP has made rapid advances recently in terms of the availability of corpora and pre-trained models, benchmark datasets on standard NLU tasks are limited.To this end, we introduce INDICXNLI, an NLI dataset for 11 Indic languages.It has been created by high-quality machine translation of the original English XNLI dataset and our analysis attests to the quality of INDICXNLI.By finetuning different pre-trained LMs on this IN-DICXNLI, we analyze various cross-lingual transfer techniques with respect to the impact of the choice of language models, languages, multi-linguality, mix-language input, etc.These experiments provide us with useful insights into the behaviour of pre-trained models for a diverse set of languages.
Continual Learning for Natural Language Generations with Transformer Calibration

Conventional natural language process (NLP) generation models are trained offline with a given dataset for a particular task, which is referred to as isolated learning.Research on sequence-to-sequence language generation aims to study continual learning model to constantly learning from sequentially encountered tasks.However, continual learning studies often suffer from catastrophic forgetting, a persistent challenge for lifelong learning.In this paper, we present a novel NLP transformer model that attempts to mitigate catastrophic forgetting in online continual learning from a new perspective, i.e., attention calibration.We model the attention in the transformer as a calibrated unit in a general formulation, where the attention calibration could give benefits to balance the stability and plasticity of continual learning algorithms through influencing both their forward inference path and backward optimization path.Our empirical experiments, paraphrase generation and dialog response generation, demonstrate that this work outperforms state-of-theart models by a considerable margin and effectively mitigate the forgetting.
Standardized Clinical Annotation of Digital Histopathology Slides at the Point of Diagnosis

<h2>Abstract</h2> As digital pathology replaces conventional glass slide microscopy as a means of reporting cellular pathology samples, the annotation of digital pathology whole slide images is rapidly becoming part of a pathologist's regular practice. Currently, there is no recognizable organization of these annotations, and as a result, pathologists adopt an arbitrary approach to defining regions of interest, leading to irregularity and inconsistency and limiting the downstream efficient use of this valuable effort. In this study, we propose a Standardized Annotation Reporting Style for digital whole slide images. We formed a list of 167 commonly annotated entities (under 12 specialty subcategories) based on review of Royal College of Pathologists and College of American Pathologists documents, feedback from reporting pathologists in our NHS department, and experience in developing annotation dictionaries for PathLAKE research projects. Each entity was assigned a suitable annotation shape, SNOMED CT (SNOMED International) code, and unique color. Additionally, as an example of how the approach could be expanded to specific tumor types, all lung tumors in the fifth World Health Organization of thoracic tumors 2021 were included. The proposed standardization of annotations increases their utility, making them identifiable at low power and searchable across and between cases. This would aid pathologists reporting and reviewing cases and enable annotations to be used for research. This structured approach could serve as the basis for an industry standard and be easily adopted to ensure maximum functionality and efficiency in the use of annotations made during routine clinical examination of digital slides.
Collaborative privacy-preserving analysis of oncological data using multiparty homomorphic encryption

Real-world healthcare data sharing is instrumental in constructing broader-based and larger clinical datasets that may improve clinical decision-making research and outcomes. Stakeholders are frequently reluctant to share their data without guaranteed patient privacy, proper protection of their datasets, and control over the usage of their data. Fully homomorphic encryption (FHE) is a cryptographic capability that can address these issues by enabling computation on encrypted data without intermediate decryptions, so the analytics results are obtained without revealing the raw data. This work presents a toolset for collaborative privacy-preserving analysis of oncological data using multiparty FHE. Our toolset supports survival analysis, logistic regression training, and several common descriptive statistics. We demonstrate using oncological datasets that the toolset achieves high accuracy and practical performance, which scales well to larger datasets. As part of this work, we propose a cryptographic protocol for interactive bootstrapping in multiparty FHE, which is of independent interest. The toolset we develop is general-purpose and can be applied to other collaborative medical and healthcare application domains.
Enhancing Email Spam Filter's Accuracy Using Machine Learning

In today's world, practically everyone uses emails on a regular basis. In our proposed research, we offer a machine learning-based technique for improving the accuracy of email spam filters. Traditional rule-based filters have become less effective as the number of spam emails has increased tremendously. Machine learning methods, particularly supervised learning, are often used to train models to determine if an email is spam or not. To achieve more accurate results when categorizing email spam, we need to build a simple and uncomplicated machine learning model. We chose the Naive Bayes strategy for our model since it is faster and more accurate than the rest of the algorithms. The recommended solution may be integrated into existing email systems to improve spam filtering capability. This review paper presents an outline of the machine learning model that we have proposed.
A Review of Central Bank Digital Currencies

This is a review paper on the exploratory research done by various central banks and international organisations on CBDCs and the framework to be followed for implementing these CBDCs so that the monetary system of a specific country remains unaffected. CBDCs and cryptocurrency in general as we know have become quite popular and are the talking point for many research scholars, economists and investment analysts. While many believe cryptocurrencies to have no intrinsic value, the value of the underlying technology i.e blockchain is something that can be argued upon. CBDCs remove the major cons of cryptocurrency by providing stability to the volatile cryptocurrency market and by acting as a legal tender thereby gaining the trust of retail public. Many governments believe that the introduction of CBDCs into the economy could simplify the payment’s structure thereby increasing the velocity of money leading to higher GDP growth. But one possibility that shatters this glass of optimistic views is the possibility of disintermediation of banks which could adversely affect the credit creation capacity of the country and disintegrate the entire monetary policy formation mechanism. Therefore, though the concept of digital currencies may excite many people, its adoption in an economy should be considered only after thorough research and pilot programs conducted by the central banks.