Reviewerly/exHarmony huggingface dataset:
An Error-Guided Correction Model for Chinese Spelling Error Correction

Although existing neural network approaches have achieved great success on Chinese spelling correction, there is still room to improve.The model is required to avoid overcorrection and to distinguish a correct token from its phonological and visually similar ones.In this paper, we propose an error-guided correction model (EGCM) to improve Chinese spelling correction.By borrowing the powerful ability of BERT, we propose a novel zeroshot error detection method to do a preliminary detection, which guides our model to attend more on the probably wrong tokens in encoding and to avoid modifying the correct tokens in generating.Furthermore, we introduce a new loss function to integrate the error confusion set, which enables our model to distinguish easily misused tokens.Moreover, our model supports highly parallel decoding to meet real application requirements.Experiments are conducted on widely used benchmarks.Our model achieves superior performance against state-ofthe-art approaches by a remarkable margin, on both the correction quality and computation speed.
AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Models

There are growing interests in adapting large-scale language models using parameterefficient fine-tuning methods.However, accelerating the model itself and achieving better inference efficiency through model compression has not been thoroughly explored yet.Model compression could provide the benefits of reducing memory footprints, enabling low-precision computations, and ultimately achieving cost-effective inference.To combine parameter-efficient adaptation and model compression, we propose AlphaTuning consisting of post-training quantization of the pre-trained language model and fine-tuning only some parts of quantized parameters for a target task.Specifically, AlphaTuning works by employing binary-coding quantization, which factorizes the full-precision parameters into binary parameters and a separate set of scaling factors.During the adaptation phase, the binary values are frozen for all tasks, while the scaling factors are fine-tuned for the downstream task.We demonstrate that AlphaTuning, when applied to GPT-2 and OPT, performs competitively with full fine-tuning on a variety of downstream tasks while achieving >10× compression ratio under 4-bit quantization and >1,000× reduction in the number of trainable parameters.
CN-AutoMIC: Distilling Chinese Commonsense Knowledge from Pretrained Language Models

Commonsense knowledge graphs (CKGs) are increasingly applied in various natural language processing tasks.However, most existing CKGs are limited to English, which hinders related research in non-English languages.Meanwhile, directly generating commonsense knowledge from pretrained language models has recently received attention, yet it has not been explored in non-English languages.In this paper, we propose a large-scale Chinese CKG generated from multilingual PLMs, named as CN-AutoMIC, aiming to fill the research gap of non-English CKGs.To improve the efficiency, we propose generate-by-category strategy to reduce invalid generation.To ensure the filtering quality, we develop cascaded filters to discard low-quality results.To further increase the diversity and density, we introduce a bootstrapping iteration process to reuse generated results.Finally, we conduct detailed analyses on CN-AutoMIC from different aspects.Empirical results show the proposed CKG has high quality and diversity, surpassing the direct translation version of similar English CKGs.We also find some interesting deficiency patterns and differences between relations, which reveal pending problems in commonsense knowledge generation.We share the resources and related models for further study 1 .
You can’t pick your neighbors, or can you? When and How to Rely on Retrieval in the kNN-LM

Retrieval-enhanced language models (LMs), which condition their predictions on text retrieved from large external datastores, have recently shown significant perplexity improvements compared to standard LMs.One such approach, the kNN-LM, interpolates any existing LM's predictions with the output of a k-nearest neighbors model and requires no additional training.In this paper, we explore the importance of lexical and semantic matching in the context of items retrieved by kNN-LM.We find two trends: (1) the presence of large overlapping n-grams between the datastore and evaluation set plays an important factor in strong performance, even when the datastore is derived from the training data; and (2) the kNN-LM is most beneficial when retrieved items have high semantic similarity with the query.Based on our analysis, we define a new formulation of the kNN-LM that uses retrieval quality to assign the interpolation coefficient.We empirically measure the effectiveness of our approach on two English language modeling datasets, Wikitext-103 and PG-19.Our re-formulation of the kNN-LM is beneficial in both cases, and leads to nearly 4% improvement in perplexity on the Wikitext-103 test set.
BioSimCSE: BioMedical Sentence Embeddings using Contrastive learning

Sentence embeddings in the form of fixed-size vectors that capture the information in the sentence as well as the context are critical components of Natural Language Processing systems.With transformer model based sentence encoders outperforming the other sentence embedding methods in the general domain, we explore the transformer based architectures to generate dense sentence embeddings in the biomedical domain.In this work, we present BioSimCSE, where we train sentence embeddings with domain specific transformer based models with biomedical texts.We assess our model's performance with zero-shot and finetuned settings on Semantic Textual Similarity (STS) and Recognizing Question Entailment (RQE) tasks.Our BioSimCSE model using Bi-oLinkBERT achieves state of the art (SOTA) performance on both tasks.
AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages

In recent years, multilingual pre-trained language models have gained prominence due to their remarkable performance on numerous downstream Natural Language Processing tasks (NLP).However, pre-training these large multilingual language models requires a lot of training data, which is not available for African Languages.Active learning is a semi-supervised learning algorithm, in which a model consistently and dynamically learns to identify the most beneficial samples to train itself on, in order to achieve better optimization and performance on downstream tasks.Furthermore, active learning effectively and practically addresses real-world data scarcity.Despite all its benefits, active learning, in the context of NLP and especially multilingual language models pretraining, has received little consideration.In this paper, we present AfroLM, a multilingual language model pretrained from scratch on 23 African languages (the largest effort to date) using our novel self-active learning framework.Pretrained on a dataset significantly (14x) smaller than existing baselines, AfroLM outperforms many multilingual pretrained language models (AfriBERTa, XLMRbase, mBERT) on various NLP downstream tasks (NER, text classification, and sentiment analysis).Additional out-of-domain sentiment analysis experiments show that AfroLM is able to generalize well across various domains.We release the code source, and our datasets used in our framework at https://github.com/bonaventuredossou/MLM_AL.
Offer a Different Perspective: Modeling the Belief Alignment of Arguments in Multi-party Debates

In contexts where debate and deliberation are the norm, the participants are regularly presented with new information that conflicts with their original beliefs.When required to update their beliefs (belief alignment), they may choose arguments that align with their worldview (confirmation bias).We test this and competing hypotheses in a constraintbased modeling approach to predict the winning arguments in multi-party interactions in the Reddit Change My View and Intelligence Squared debates datasets.We adopt a hierarchical generative Variational Autoencoder as our model and impose structural constraints that reflect competing hypotheses about the nature of argumentation.Our findings suggest that in most settings, predictive models that anticipate winning arguments to be further from the initial argument of the opinion holder are more likely to succeed.
Reorder and then Parse, Fast and Accurate Discontinuous Constituency Parsing

Discontinuous constituency parsing is still kept developing for its efficiency and accuracy are far behind its continuous counterparts.Motivated by the observation that a discontinuous constituent tree can be simply transformed into a pseudo-continuous one by artificially reordering words in the sentence, we propose a novel reordering method, thereby construct fast and accurate discontinuous constituency parsing systems working in continuous way.Specifically, we model the relative position changes of words as a list of actions.By parsing and performing this actions, the corresponding pseudocontinuous sequence is derived.Discontinuous parse tree can be further inferred via integrating a high-performance pseudo-continuous constituency parser.Our systems are evaluated on three classical discontinuous constituency treebanks, achieving new state-of-the-art on two treebanks and showing a distinct advantage in speed.
Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study

This paper presents an empirical study to build relation extraction systems in low-resource settings.Based upon recent pre-trained language models, we comprehensively investigate three schemes to evaluate the performance in lowresource settings: (i) different types of promptbased methods with few-shot labeled data; (ii) diverse balancing methods to address the longtailed distribution issue; (iii) data augmentation technologies and self-training to generate more labeled in-domain data.We create a benchmark with 8 relation extraction (RE) datasets covering different languages, domains and contexts and perform extensive comparisons over the proposed schemes with combinations.Our experiments illustrate: (i) Though prompt-based tuning is beneficial in low-resource RE, there is still much potential for improvement, especially in extracting relations from cross-sentence contexts with multiple relational triples; (ii) Balancing methods are not always helpful for RE with longtailed distribution; (iii) Data augmentation complements existing baselines and can bring much performance gain, while self-training may not consistently achieve advancement to low-resource RE 1 .
Con-NAT: Contrastive Non-autoregressive Neural Machine Translation

Inspired by the success of contrastive learning in natural language processing, we incorporate contrastive learning into the conditional masked language model which is extensively used in non-autoregressive neural machine translation (NAT).Accordingly, we propose a Contrastive Non-autoregressive Neural Machine Translation (Con-NAT) model.Con-NAT optimizes the similarity of several different representations of the same token in the same sentence.We propose two methods to obtain various representations: Contrastive Common Mask and Contrastive Dropout.Positive pairs are various different representations of the same token, while negative pairs are representations of different tokens.In the feature space, the model with contrastive loss pulls positive pairs together and pushes negative pairs away.We conduct extensive experiments on six translation directions with different data sizes.The results demonstrate that Con-NAT showed a consistent and significant improvement in fully and iterative NAT.Con-NAT is state-of-the-art on .
APEACH: Attacking Pejorative Expressions with Analysis on Crowd-Generated Hate Speech Evaluation Datasets

In hate speech detection, developing training and evaluation datasets across various domains is the critical issue.Whereas, major approaches crawl social media texts and hire crowd-workers to annotate the data.Following this convention often restricts the scope of pejorative expressions to a single domain lacking generalization.Sometimes domain overlap between training corpus and evaluation set overestimate the prediction performance when pretraining language models on low-data language.To alleviate these problems in Korean, we propose APEACH that asks unspecified users to generate hate speech examples followed by minimal post-labeling.We find that APEACH can collect useful datasets that are less sensitive to the lexical overlaps between the pretraining corpus and the evaluation set, thereby properly measuring the model performance.
Learning Better Intent Representations for Financial Open Intent Classification

With the recent surge of NLP technologies in the financial domain, banks and other financial entities have adopted virtual agents (VA) to assist customers.A challenging problem for VAs in this domain is determining a user's reason or intent for contacting the VA, especially when the intent was unseen or open during the VA's training.One method for handling open intents is adaptive decision boundary (ADB) post-processing, which learns tight decision boundaries from intent representations to separate known and open intents.We propose incorporating two methods for supervised pre-training of intent representations: prefixtuning and fine-tuning just the last layer of a large language model (LLM).With this proposal, our accuracy is 1.63% -2.07%higher than the prior state-of-the-art ADB method for open intent classification on the bank-ing77 benchmark amongst others.Notably, we only supplement the original ADB model with 0.1% additional trainable parameters.Ablation studies also determine that our method yields better results than full fine-tuning the entire model.We hypothesize that our findings could stimulate a new optimal method of downstream tuning that combines parameter efficient tuning modules with fine-tuning a subset of the base model's layers.
Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics

Recent works that revealed the vulnerability of dialogue state tracking (DST) models to distributional shifts have made holistic comparisons on robustness and qualitative analyses increasingly important for understanding their relative performance.We present our findings from standardized and comprehensive DST diagnoses, which have previously been sparse and uncoordinated, using our toolkit, CheckDST, a collection of robustness tests and failure mode analytics.We discover that different classes of DST models have clear strengths and weaknesses, where generation models are more promising for handling language variety while span-based classification models are more robust to unseen entities.Prompted by this discovery, we also compare checkpoints from the same model and find that the standard practice of selecting checkpoints using validation loss/accuracy is prone to overfitting and each model class has distinct patterns of failure.Lastly, we demonstrate how our diagnoses motivate a pre-finetuning procedure with non-dialogue data that offers comprehensive improvements to generation models by alleviating the impact of distributional shifts through transfer learning.
Confidence-aware Sentiment Quantification via Sentiment Perturbation Modeling

Sentiment Quantification aims to detect the overall sentiment polarity of users from a set of reviews corresponding to a target. Existing methods equally treat and aggregate individual reviews' sentiment to judge the overall sentiment polarity. However, the confidence of each review is not equal in sentiment quantification where sentiment perturbation arising from high- and low-confidence reviews may degrade the accuracy of Sentiment Quantification. Specifically, fake reviews with deceptive sentiments are low confidence, which perturbs the overall sentiment prediction. Whereas, some reviews generated by responsible users are high confidence. They contain authoritative suggestions so they should be emphasized in Sentiment Quantification. In this paper, we design and build COSE, a confidence-aware sentiment quantification framework, which can measure the confidence of individual reviews to eliminate sentiment perturbation and facilitate sentiment quantification. We design a Review Graph that achieves review confidence modeling in an unsupervised manner and obtains review confidence representations. Moreover, we develop a dynamic fusion attention mechanism, which produces sentiment “de-perturbation” vectors to eliminate the sentiment perturbation based on the confidence representations. Extensive experiments on large-scale review datasets validate the significant superiority of COSE over the state-of-the-art.
Machine Learning Alternatives to Response Surface Models

In the Design of Experiments, we seek to relate response variables to explanatory factors. Response Surface methodology (RSM) approximates the relation between output variables and a polynomial transform of the explanatory variables using a linear model. Some researchers have tried to adjust other types of models, mainly nonlinear and nonparametric. We present a large panel of Machine Learning approaches that may be good alternatives to the classical RSM approximation. The state of the art of such approaches is given, including classification and regression trees, ensemble methods, support vector machines, neural networks and also direct multi-output approaches. We survey the subject and illustrate the use of ten such approaches using simulations and a real use case. In our simulations, the underlying model is linear in the explanatory factors for one response and nonlinear for the others. We focus on the advantages and disadvantages of the different approaches and show how their hyperparameters may be tuned. Our simulations show that even when the underlying relation between the response and the explanatory variables is linear, the RSM approach is outperformed by the direct neural network multivariate model, for any sample size (&lt;50) and much more for very small samples (15 or 20). When the underlying relation is nonlinear, the RSM approach is outperformed by most of the machine learning approaches for small samples (n ≤ 30).
CADUCEO: A Platform to Support Federated Healthcare Facilities through Artificial Intelligence

Data-driven algorithms have proven to be effective for a variety of medical tasks, including disease categorization and prediction, personalized medicine design, and imaging diagnostics. Although their performance is frequently on par with that of clinicians, their widespread use is constrained by a number of obstacles, including the requirement for high-quality data that are typical of the population, the difficulty of explaining how they operate, and ethical and regulatory concerns. The use of data augmentation and synthetic data generation methodologies, such as federated learning and explainable artificial intelligence ones, could provide a viable solution to the current issues, facilitating the widespread application of artificial intelligence algorithms in the clinical application domain and reducing the time needed for prevention, diagnosis, and prognosis by up to 70%. To this end, a novel AI-based functional framework is conceived and presented in this paper.
Sentiment Analysis of Hybrid Network Model Based on Attention

The existing text sentiment analysis models based on deep learning and neural network usually have problems such as incomplete text feature extraction and failure to consider the impact of key information on text sentiment tendency. Based on the parallel hybrid network and the two-way attention mechanism, an improved text sentiment analysis model is proposed. The model first takes the word vector trained by the BERT language model as the input, and then extracts the global and local features of the context simultaneously through the parallel hybrid neural network constructed by the Convolution Neural Network (CNN) and The Bidirectional Gated Recurrent Unit (BiGRU), so as to improve the feature extraction ability of the model. It also integrates the dual-way attention mechanism to strengthen the key information in the global feature and local feature, and the feature vectors obtained by feature fusion are used for sentiment analysis.
APPLICATION OF SPECTRAL METHODS OF BOUNDARY INTEGRAL EQUATIONS FOR MODELING OF NANOOPTICAL DEVICES

Efficiency of modeling of optical nanostructures depends not only on the accuracy of the description of the new physical processes which appear in the new configurations of resonantly scattering and resonantly absorbing structures, but also on the selection of appropriate algorithms for solving the corresponding mathematical problem and numerical parameters of modeling depending on parameters of elements. This is why solving complicated problems of modeling complex resonantly scattering and resonantly absorbing electrodynamic nanostructures involves deep learning of all groups of new unknown effects. In this work a semi-analytical algorithm is developed based on parametrized by conformal mapping techniques spectral method of boundary integral equations with analytical regularization based on singularity subtraction enhanced by Fast Fourier transform, that contrary to the classical schemes, which are based on finite difference and finite element methods, allows to take into account the complex-valued functional dependence of dielectric permittivity of plasmonic materials on the wavelength (even when its value is tabulated) and to solve the problems with static and dynamic singularities in integral equations. Due to sensibility of plasmon resonances to changes in external medium such nanostructures are used in modern medicine, pharmacy and also as chemical and biological sensors. In this work main efforts are directed on generation of algorithm for investigation of dielectric nanostructures with static singularities.
The Effect of Digital Marketing Implementation Towards Muslim Fashion Brand Awareness and Brand Image on Covid’19 Pandemic

Digital marketing has a positive impact on brand awareness and brand image. This can have a good effect on the economy as it can increase brand awareness and brand image of Muslim fashion brands. This good economic effect will translate into improved company performance in covid’19. This study aims to determine the influence of the company’s digital marketing on brand awareness and brand image on covid’19 in Muslim fashion brands. This study uses multiple regression on panel data from a sample of 100 respondents based in Indonesia. The variable of this research is digital marketing in Muslim fashion brands. Other variables are brand awareness and brand image to see what effect digital marketing has on these variables in covid’19. The results of the study resulted in t value &gt; t table and sig &lt; 0.05 from each hypothesis that everything was declared accepted because t value &gt; t table and sig &lt; 0.05. This study shows a significant influence of digital marketing on Muslim fashion brands on brand awareness and brand image during the covid’19 pandemic.