Reviewerly/exHarmony huggingface dataset:
When classifying grammatical role, BERT doesn’t care about word order... except when it matters

Because meaning can often be inferred from lexical semantics alone, word order is often a redundant cue in natural language.For example, the words chopped, chef, and onion are more likely used to convey "The chef chopped the onion," not "The onion chopped the chef."Recent work has shown large language models to be surprisingly word order invariant, but crucially has largely considered natural prototypical inputs, where compositional meaning mostly matches lexical expectations.To overcome this confound, we probe grammatical role representation in English BERT and GPT-2, on instances where lexical expectations are not sufficient, and word order knowledge is necessary for correct classification.Such nonprototypical instances are naturally occurring English sentences with inanimate subjects or animate objects, or sentences where we systematically swap the arguments to make sentences like "The onion chopped the chef".We find that, while early layer embeddings are largely lexical, word order is in fact crucial in defining the later-layer representations of words in semantically non-prototypical positions.Our experiments isolate the effect of word order on the contextualization process, and highlight how models use context in the uncommon, but critical, instances where it matters.
Sammaan@LT-EDI-ACL2022: Ensembled Transformers Against Homophobia and Transphobia

Hateful and offensive content on social media platforms can have negative effects on users and can make online communities more hostile towards certain people and hamper equality, diversity and inclusion.In this paper, we describe our approach to classify homophobia and transphobia in social media comments.We used an ensemble of transformer based models to build our classifier.Our model ranked 2nd for English, 8th for Tamil and 10th for Tamil-English.
Local Structure Matters Most: Perturbation Study in NLU

Recent research analyzing the sensitivity of natural language understanding models to wordorder perturbations has shown that neural models are surprisingly insensitive to the order of words.In this paper, we investigate this phenomenon by developing order-altering perturbations on the order of words, subwords, and characters to analyze their effect on neural models' performance on language understanding tasks.We experiment with measuring the impact of perturbations to the local neighborhood of characters and global position of characters in the perturbed texts and observe that perturbation functions found in prior literature only affect the global ordering while the local ordering remains relatively unperturbed.We empirically show that neural models, invariant of their inductive biases, pretraining scheme, or the choice of tokenization, mostly rely on the local structure of text to build understanding and make limited use of the global structure. Related WorkImportance of Syntax Discussions on semantics (Culbertson and Adger, 2014;Futrell et al., 2020) agree on specific orders of words to be necessary for comprehending text.Psycholinguistic research (Hale, 2017) corroborates this through evaluating sentence comprehension mechanisms of humans.Hence, interpreting language as a bag-of-words could limit the expressions conveyed through the word-orders (Harris, 1954;Le and Mikolov, 2014) and understanding syntax 2 becomes an essential artifact.Recently, Mollica et al. (2020) found that humans were robust to wordordering perturbations in text as long as local ordering of text was roughly preserved.Prior works have explored the relationship between neural models and syntax.Goldberg (2019); Hewitt and Manning (2019) both show that BERT (Devlin et al., 2019) models have some syntactic capacity.Lin et al. (2019b) show that BERT represents information hierarchically and concludes that BERT models linguistically relevant aspects in a hierarchical structure.Tenney et al. (2019); Liu et al. (2019b) show that the contextual embeddings that BERT outputs contain syntactic information that could be used in downstream tasks.While it seems that syntax is both important, and to an extent, understood by the recent family of PT models, it is unclear how much use they make of it.Glavaš and Vulić (2020) showed that pretraining BERT on syntax does not seem to improve downstream performance much.Warstadt et al. (2020) showed that while models such as BERT do understand syntax, they often prefer not to use that
Exploring nexus among big data analytic capability and organizational performance through mediation of supply chain agility

The organization needs to improve on a steady basis. For this purpose, organizations must gauge their performance regularly. To achieve this purpose, the agility of the supply chain may play a key role. Therefore, this study was designed to explore the relationship between big data analytics, organizational flexibility, supply chain agility, and organizational performance. This study assessed the mediation effect of supply chain agility as well. The research design of the cross-sectional and research approach was quantitative. The data of this study was gathered from the retail sector employees. In total, 516 questionnaires were distributed using simple random sampling. The usable response rate of the study was 54.90%. The gathered data was examined through smart PLS 3.3.2. The findings of the study revealed that Supply chain agility plays a crucial role in improving the performance of the organization. The study also confirmed the mediating effect of supply chain agility. The findings of the study are helpful for the policymakers of the retail sector.
An Anomaly Detection Approach for Plankton Species Discovery

Plankton is one of the most abundant and diverse class of microscopic organisms inhabiting the Earth. Their enormous intra- and inter-species genetic and phenotypic diversity, coupled with the limited amount of large survey data, makes it hard to obtain a complete representation of this important class of organisms. Hence, the classification accuracy of novel supervised machine learning algorithms is bound to be limited by the incompleteness of the training data. In this work we introduce an efficient pipeline centered around a novel anomaly detection algorithm to discover and classify new plankton species, in situ, with the aim of automatically populating a plankton database in an unsupervised fashion. Our pipeline utilizes the concept of anomaly detection to separate a novel species from the ones contained in an initial existing database. Our results show that the implemented algorithm outperforms four state-of-the-art methods for outlier detection on the plankton dataset used in our analysis. Finally, using a leave-one-out approach, we prove that our pipeline is able to identify unknown plankton species with high-accuracy.
Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings

Style transfer is the task of rewriting a sentence into a target style while approximately preserving content.While most prior literature assumes access to a large style-labelled corpus, recent work (Riley et al., 2021) has attempted "few-shot" style transfer using just 3-10 sentences at inference for style extraction.In this work, we study a relevant low-resource setting: style transfer for languages where no style-labelled corpora are available.We notice that existing few-shot methods perform this task poorly, often copying inputs verbatim.We push the state-of-the-art for few-shot style transfer with a new method modeling the stylistic difference between paraphrases.When compared to prior work, our model achieves 2-3x better performance in formality transfer and code-mixing addition across seven languages.Moreover, our method is better at controlling the style transfer magnitude using an input scalar knob.We report promising qualitative results for several attribute transfer tasks (sentiment transfer, simplification, gender neutralization, text anonymization) all without retraining the model.Finally, we find model evaluation to be difficult due to the lack of datasets and metrics for many languages.To facilitate future research we crowdsource formality annotations for 4000 sentence pairs in four Indic languages, and use this data to design our automatic evaluations. 1
Simple Semantic-based Data Augmentation for Named Entity Recognition in Biomedical Texts

Data augmentation is important in addressing data sparsity and low resources in NLP.Unlike data augmentation for other tasks such as sentence-level and sentence-pair ones, data augmentation for named entity recognition (NER) requires preserving the semantic of entities.To that end, in this paper we propose a simple semantic-based data augmentation method for biomedical NER.Our method leverages semantic information from pre-trained language models for both entity-level and sentence-level.Experimental results on two datasets: i2b2-2010 (English) and VietBioNER (Vietnamese) showed that the proposed method could improve NER performance.
Rendering Secure and Trustworthy Edge Intelligence in 5G-Enabled IIoT Using Proof of Learning Consensus Protocol

Industrial Internet of Things (IIoT) and fifth generation (5G) network have fueled the development of Industry 4.0 by providing an unparalleled connectivity and intelligence to ensure timely (or real time) and optimal decision-making. Under this umbrella, the edge intelligence is ready to propel another ripple in the industrial growth by ensuring the next generation of connectivity and performance. With the recent proliferation of blockchain, edge intelligence enters a new era, where each edge trains the local learning model, then interconnecting the whole learning models in a distributed blockchain manner, known as blockchain-assisted federated learning. However, it is quiet challenging task to provide secure edge intelligence in 5G-enabled IIoT environment alongside ensuring latency and throughput. In this article, we propose a proof-of-learning consensus protocol that considers the reputation opinion for edge blockchain to ensure secure and trustworthy edge intelligence in IIoT. This protocol fetches each edge’s reputation opinion by executing a smart contract, and partly adopts the winner’s learning model according to its reputation opinion. By quantitative performance analysis and simulation experiments, the proposed scheme demonstrates the superior performance in contrast to the traditional counterparts.
Semantic Segmentation of Text Using Deep Learning

Given a text, can we segment it into semantically coherent sections in an automatic way? Can we detect the semantic boundaries, if we know how many they are? Can we determine how many semantically distinct sections are in the text? These are the questions we address in this paper. To respond, we use the Bidirectional Encoder Representation from Transformer (BERT) to analyze the text and evaluate a function that we call local incoherence, which we expect to show maxima at the points where a semantic boundary is detected. Our results, although preliminary, are encouraging and suggest that our approach can be successfully applied. However, they are quite sensitive with respect to the text quality, as it happens in the case in which the text is derived from an audio stream via Automatic Speech Recognition techniques.
Knowledge Distillation Meets Few-Shot Learning: An Approach for Few-Shot Intent Classification Within and Across Domains

Large Transformer-based natural language understanding models have achieved state-of-theart performance in dialogue systems.However, scarce labeled data for training, the large model size, and low inference speed hinder their deployment in low-resource scenarios.Few-shot learning and knowledge distillation techniques have been introduced to reduce the need for labeled data and computational resources, respectively.However, these techniques are incompatible because few-shot learning trains models using few data, whereas, knowledge distillation requires sufficient data to train smaller, yet competitive models that run on limited computational resources.In this paper, we address the problem of distilling generalizable small models under the few-shot setting for the intent classification task.Considering in-domain and cross-domain few-shot learning scenarios, we introduce an approach for distilling small models that generalize to new intent classes and domains using only a handful of labeled examples.We conduct experiments on public intent classification benchmarks, and observe a slight performance gap between small models and large models.Overall, our results in both few-shot scenarios confirm the generalization ability of the small distilled models while having lower computational costs.
Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

Although contextualized embeddings generated from large-scale pre-trained models perform well in many tasks, traditional static embeddings (e.g., Skip-gram, Word2Vec) still play an important role in low-resource and lightweight settings due to their low computational cost, ease of deployment, and stability.In this paper, we aim to improve word embeddings by 1) incorporating more contextual information from existing pre-trained models into the Skip-gram framework, which we call Context-to-Vec; 2) proposing a post-processing retrofitting method for static embeddings independent of training by employing priori synonym knowledge and weighted vector distribution.Through extrinsic and intrinsic tasks, our methods are well proven to outperform the baselines by a large margin.
Document-Level Event Argument Extraction via Optimal Transport

Event Argument Extraction (EAE) is one of the sub-tasks of event extraction, aiming to recognize the role of each entity mention toward a specific event trigger.Despite the success of prior works in sentence-level EAE, the document-level setting is less explored.In particular, whereas syntactic structures of sentences have been shown to be effective for sentence-level EAE, prior document-level EAE models totally ignore syntactic structures for documents.Hence, in this work, we study the importance of syntactic structures in document-level EAE.Specifically, we propose to employ Optimal Transport (OT) to induce structures of documents based on sentencelevel syntactic structures and tailored to EAE task.Furthermore, we propose a novel regularization technique to explicitly constrain the contributions of unrelated context words in the final prediction for EAE.We perform extensive experiments on the benchmark documentlevel EAE dataset RAMS that leads to the state-of-the-art performance.Moreover, our experiments on the ACE 2005 dataset reveals the effectiveness of the proposed model in the sentence-level EAE by establishing new stateof-the-art results.
Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages

Pre-trained multilingual language models such as mBERT and XLM-R have demonstrated great potential for zero-shot cross-lingual transfer to low web-resource languages (LRL).However, due to limited model capacity, the large difference in the sizes of available monolingual corpora between high web-resource languages (HRL) and LRLs does not provide enough scope of co-embedding the LRL with the HRL, thereby affecting the downstream task performance of LRLs.In this paper, we argue that relatedness among languages in a language family along the dimension of lexical overlap may be leveraged to overcome some of the corpora limitations of LRLs.We propose Overlap BPE (OBPE), a simple yet effective modification to the BPE vocabulary generation algorithm which enhances overlap across related languages.Through extensive experiments on multiple NLP tasks and datasets, we observe that OBPE generates a vocabulary that increases the representation of LRLs via tokens shared with HRLs.This results in improved zero-shot transfer from related HRLs to LRLs without reducing HRL representation and accuracy.Unlike previous studies that dismissed the importance of token-overlap, we show that in the low-resource related language setting, token overlap matters.Synthetically reducing the overlap to zero can cause as much as a four-fold drop in zero-shot transfer accuracy.
Augmenting Document Representations for Dense Retrieval with Interpolation and Perturbation

Dense retrieval models, which aim at retrieving the most relevant document for an input query on a dense representation space, have gained considerable attention for their remarkable success.Yet, dense models require a vast amount of labeled training data for notable performance, whereas it is often challenging to acquire query-document pairs annotated by humans.To tackle this problem, we propose a simple but effective Document Augmentation for dense Retrieval (DAR) framework, which augments the representations of documents with their interpolation and perturbation.We validate the performance of DAR on retrieval tasks with two benchmark datasets, showing that the proposed DAR significantly outperforms relevant baselines on the dense retrieval of both the labeled and unlabeled documents.
Unsupervised multiple-choice question generation for out-of-domain Q&amp;A fine-tuning

Pre-trained models have shown very good performances on a number of question answering benchmarks especially when fine-tuned on multiple question answering datasets at once.In this work, we propose an approach for generating a fine-tuning dataset thanks to a rule-based algorithm that generates questions and answers from unannotated sentences.We show that the state-of-the-art model UnifiedQA can greatly benefit from such a system on a multiple-choice benchmark about physics, biology and chemistry it has never been trained on.We further show that improved performances may be obtained by selecting the most challenging distractors (wrong answers), with a dedicated ranker based on a pretrained RoBERTa model.
Task2Dial: A Novel Task and Dataset for Commonsense-enhanced Task-based Dialogue Grounded in Documents

This paper proposes a novel task on commonsense-enhanced task-based dialogue grounded in documents and describes the Task2Dial dataset, a novel dataset of document-grounded task-based dialogues, where an Information Giver (IG) provides instructions (by consulting a document) to an Information Follower (IF), so that the latter can successfully complete the task.In this unique setting, the IF can ask clarification questions which may not be grounded in the underlying document and require commonsense knowledge to be answered.The Task2Dial dataset poses new challenges:(1) its human reference texts show more lexical richness and variation than other document-grounded dialogue datasets; (2) generating from this set requires paraphrasing as instructional responses might have been modified from the underlying document; (3) requires commonsense knowledge, since questions might not necessarily be grounded in the document; (4) generating requires planning based on context, as task steps need to be provided in order.The Task2Dial dataset contains dialogues with an average 18.15 number of turns and 19.79 tokens per turn, as compared to 12.94 and 12 respectively in existing datasets.As such, learning from this dataset promises more natural, varied and less template-like system utterances.
Improving Chinese Grammatical Error Detection via Data augmentation by Conditional Error Generation

Chinese Grammatical Error Detection(CGED) aims at detecting grammatical errors in Chinese texts.One of the main challenges for CGED is the lack of annotated data.To alleviate this problem, previous studies proposed various methods to automatically generate more training samples, which can be roughly categorized into rule-based methods and model-based methods.The rule-based methods construct erroneous sentences by directly introducing noises into original sentences.However, the introduced noises are usually context-independent, which are quite different from those made by humans.The model-based methods utilize generative models to imitate human errors.The generative model may bring too many changes to the original sentences and generate semantically ambiguous sentences, so it is difficult to detect grammatical errors in these generated sentences.In addition, generated sentences may be error-free and thus become noisy data.To handle these problems, we propose CNEG, a novel Conditional Non-Autoregressive Error Generation model for generating Chinese grammatical errors.Specifically, in order to generate a context-dependent error, we first mask a span in a correct text, then predict an erroneous span conditioned on both the masked text and the correct span.Furthermore, we filter out error-free spans by measuring their perplexities in the original sentences.Experimental results show that our proposed method achieves better performance than all compared data augmentation methods on the CGED-2018 and CGED-2020 benchmarks.
Coherence boosting: When your pretrained language model is not paying enough attention

Long-range semantic coherence remains a challenge in automatic language generation and understanding.We demonstrate that large language models have insufficiently learned the effect of distant words on next-token prediction.We present coherence boosting, an inference procedure that increases a LM's focus on a long context.We show the benefits of coherence boosting with pretrained models by distributional analyses of generated ordinary text and dialog responses.It is also found that coherence boosting with state-of-the-art models for various zero-shot NLP tasks yields performance gains with no additional training.A: I'm Natasha.I study neural language models and dialog systems.Are you an AI researcher too?B: No, though I do like chatting with bots and laughing at their mistakes.But what was your name again?A: Oh, you forgot already?My name
As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioning

Omission and addition of content is a typical issue in neural machine translation.We propose a method for detecting such phenomena with off-the-shelf translation models.Using contrastive conditioning, we compare the likelihood of a full sequence under a translation model to the likelihood of its parts, given the corresponding source or target sequence.This allows to pinpoint superfluous words in the translation and untranslated words in the source even in the absence of a reference translation.The accuracy of our method is comparable to a supervised method that requires a custom quality estimation model.
Multi-Party Empathetic Dialogue Generation: A New Task for Dialog Systems

Empathetic dialogue assembles emotion understanding, feeling projection, and appropriate response generation.Existing work for empathetic dialogue generation concentrates on the two-party conversation scenario.Multiparty dialogues, however, are pervasive in reality.Furthermore, emotion and sensibility are typically confused; a refined empathy analysis is needed for comprehending fragile and nuanced human feelings.We address these issues by proposing a novel task called Multi-Party Empathetic Dialogue Generation in this study.Additionally, a Static-Dynamic model for Multi-Party Empathetic Dialogue Generation, SDMPED, is introduced as a baseline by exploring the static sensibility and dynamic emotion for the multi-party empathetic dialogue learning, the aspects that help SDMPED achieve the state-of-the-art performance.