Reviewerly/exHarmony huggingface dataset:
Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation

End-to-end Speech Translation (ST) aims at translating the source language speech into target language text without generating the intermediate transcriptions.However, the training of end-to-end methods relies on parallel ST data, which are difficult and expensive to obtain.Fortunately, the supervised data for automatic speech recognition (ASR) and machine translation (MT) are usually more accessible, making zero-shot speech translation a potential direction.Existing zero-shot methods fail to align the two modalities of speech and text into a shared semantic space, resulting in much worse performance compared to the supervised ST methods.In order to enable zero-shot ST, we propose a novel Discrete Cross-Modal Alignment (DCMA) method that employs a shared discrete vocabulary space to accommodate and match both modalities of speech and text.Specifically, we introduce a vector quantization module to discretize the continuous representations of speech and text into a finite set of virtual tokens, and use ASR data to map corresponding speech and text to the same virtual token in a shared codebook.This way, source language speech can be embedded in the same semantic space as the source language text, which can be then transformed into target language text with an MT module.Experiments on multiple language pairs demonstrate that our zero-shot ST method significantly improves the SOTA, and even performs on par with the strong supervised ST baselines 1 .
ReCo: Reliable Causal Chain Reasoning via Structural Causal Recurrent Neural Networks

Causal chain reasoning (CCR) is an essential ability for many decision-making AI systems, which requires the model to build reliable causal chains by connecting causal pairs.However, CCR suffers from two main transitive problems: threshold effect and scene drift.In other words, the causal pairs to be spliced may have a conflicting threshold boundary or scenario.To address these issues, we propose a novel Reliable Causal chain reasoning framework (ReCo), which introduces exogenous variables to represent the threshold and scene factors of each causal pair within the causal chain, and estimates the threshold and scene contradictions across exogenous variables via structural causal recurrent neural networks (SRNN).Experiments show that ReCo outperforms a series of strong baselines on both Chinese and English CCR datasets.Moreover, by injecting reliable causal chain knowledge distilled by ReCo, BERT can achieve better performances on four downstream causal-related tasks than BERT models enhanced by other kinds of knowledge.
FastClass: A Time-Efficient Approach to Weakly-Supervised Text Classification

Weakly-supervised text classification aims to train a classifier using only class descriptions and unlabeled data.Recent research shows that keyword-driven methods can achieve state-ofthe-art performance on various tasks.However, these methods not only rely on carefullycrafted class descriptions to obtain classspecific keywords but also require substantial amount of unlabeled data and takes a long time to train.This paper proposes FastClass, an efficient weakly-supervised classification approach.It uses dense text representation to retrieve class-relevant documents from external unlabeled corpus and selects an optimal subset to train a classifier.Compared to keyworddriven methods, our approach is less reliant on initial class descriptions as it no longer needs to expand each class description into a set of class-specific keywords.Experiments on a wide range of classification tasks show that the proposed approach frequently outperforms keyword-driven models in terms of classification accuracy and often enjoys orders-ofmagnitude faster training speed.
Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialogues

The construction of open-domain dialogue systems requires high-quality dialogue datasets.The dialogue data admits a wide variety of responses for a given dialogue history, especially responses with different semantics.However, collecting high-quality such a dataset in most scenarios is labor-intensive and timeconsuming.In this paper, we propose a data augmentation method to automatically augment high-quality responses with different semantics by counterfactual inference.Specifically, given an observed dialogue, our counterfactual generation model first infers semantically different responses by replacing the observed reply perspective with substituted ones.Furthermore, our data selection method filters out detrimental augmented responses.Experimental results show that our data augmentation method can augment high-quality responses with different semantics for a given dialogue history, and can outperform competitive baselines on multiple downstream tasks.
Norm-based Noisy Corpora Filtering and Refurbishing in Neural Machine Translation

Recent advances in neural machine translation depend on massive parallel corpora, which are collected from any open source without much guarantee of quality.It stresses the need for noisy corpora filtering, but existing methods are insufficient to solve this issue.They spend much time ensembling multiple scorers trained on clean bitexts, unavailable for low-resource languages in practice.In this paper, we propose a norm-based noisy corpora filtering and refurbishing method with no external data and costly scorers.The noisy and clean samples are separated based on how much information from the source and target sides the model requires to fit the given translation.For the unparallel sentence, the target-side history translation is much more important than the source context, contrary to the parallel ones.The amount of these two information flows can be measured by norms of source-/target-side context vectors.Moreover, we propose to reuse the discovered noisy data by generating pseudo labels via online knowledge distillation.Extensive experiments show that our proposed filtering method performs comparably with state-ofthe-art noisy corpora filtering techniques but is more efficient and easier to operate.Noisy sample refurbishing further enhances the performance by making the most of the given data 1 .
AX-MABSA: A Framework for Extremely Weakly Supervised Multi-label Aspect Based Sentiment Analysis

Aspect Based Sentiment Analysis is a dominant research area with potential applications in social media analytics, business, finance, and health.Prior works in this area are primarily based on supervised methods, with a few techniques using weak supervision limited to predicting a single aspect category per review sentence.In this paper, we present an extremely weakly supervised multi-label Aspect Category Sentiment Analysis framework which does not use any labelled data.We only rely on a single word per class as an initial indicative information.We further propose an automatic word selection technique to choose these seed categories and sentiment words.We explore unsupervised language model post-training to improve the overall performance, and propose a multi-label generator model to generate multiple aspect category-sentiment pairs per review sentence.Experiments conducted on four benchmark datasets showcase our method to outperform other weakly supervised baselines by a significant margin. 1
Azimuth: Systematic Error Analysis for Text Classification

Gabrielle Gauthier-melancon, Orlando Marquez Ayala, Lindsay Brin, Chris Tyler, Frederic Branchaud-charron, Joseph Marinier, Karine Grande, Di Le. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2022.
Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks

Teacher-student knowledge distillation is a popular technique for compressing today's prevailing large language models into manageable sizes that fit low-latency downstream applications.Both the teacher and the choice of transfer set used for distillation are crucial ingredients in creating a high quality student.Yet, the generic corpora used to pretrain the teacher and the corpora associated with the downstream target domain are often significantly different, which raises a natural question: should the student be distilled over the generic corpora, so as to learn from highquality teacher predictions, or over the downstream task corpora to align with finetuning?Our study investigates this trade-off using Domain Classification (DC) and Intent Classification/Named Entity Recognition (ICNER) as downstream tasks.We distill several multilingual students from a larger multilingual LM with varying proportions of generic and taskspecific datasets, and report their performance after finetuning on DC and ICNER.We observe significant improvements across tasks and test sets when only task-specific corpora is used.We also report on how the impact of adding task-specific data to the transfer set correlates with the similarity between generic and task-specific data.Our results clearly indicate that, while distillation from a generic LM benefits downstream tasks, students learn better using target domain data even if it comes at the price of noisier teacher predictions.In other words, target domain data still trumps teacher knowledge.
Automatic Scene-based Topic Channel Construction System for E-Commerce

Scene marketing that well demonstrates user interests within a certain scenario has proved effective for offline shopping.To conduct scene marketing for e-commerce platforms, this work presents a novel product form, scene-based topic channel which typically consists of a list of diverse products belonging to the same usage scenario and a topic title that describes the scenario with marketing words.As manual construction of channels is time-consuming due to billions of products as well as dynamic and diverse customers' interests, it is necessary to leverage AI techniques to automatically construct channels for certain usage scenarios and even discover novel topics.To be specific, we first frame the channel construction task as a two-step problem, i.e., scene-based topic generation and product clustering, and propose an E-commerce Scene-based Topic Channel construction system (i.e., ESTC) to achieve automated production, consisting of scene-based topic generation model for the e-commerce domain, product clustering on the basis of topic similarity, as well as quality control based on automatic model filtering and human screening.Extensive offline experiments and online A/B test validates the effectiveness of such a novel product form as well as the proposed system.In addition, we also introduce the experience of deploying the proposed system on a real-world e-commerce recommendation platform.
Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions

As multimodal learning finds applications in a wide variety of high-stakes societal tasks, investigating their robustness becomes important.Existing work has focused on understanding the robustness of vision-and-language models to imperceptible variations on benchmark tasks.In this work, we investigate the robustness of multimodal classifiers to cross-modal dilutions -a plausible variation.We develop a model that, given a multimodal (image + text) input, generates additional dilution text that (a) maintains relevance and topical coherence with the image and existing text, and (b) when added to the original text, leads to misclassification of the multimodal input.Via experiments on Crisis Humanitarianism and Sentiment Detection tasks, we find that the performance of task-specific fusion-based multimodal classifiers drops by 23.3% and 22.5%, respectively, in the presence of dilutions generated by our model.Metric-based comparisons with several baselines and human evaluations indicate that our dilutions show higher relevance and topical coherence, while simultaneously being more effective at demonstrating the brittleness of the multimodal classifiers.Our work aims to highlight and encourage further research on the robustness of deep multimodal models to realistic variations, especially in human-facing societal applications.
Named Entity Recognition in Industrial Tables using Tabular Language Models

Specialized transformer-based models for encoding tabular data have gained interest in academia.Although tabular data is omnipresent in industry, applications of table transformers are still missing.In this paper, we study how these models can be applied to an industrial Named Entity Recognition (NER) problem where the entities are mentioned in tabularstructured spreadsheets.The highly technical nature of spreadsheets as well as the lack of labeled data present major challenges for finetuning transformer-based models.Therefore, we develop a dedicated table data augmentation strategy based on available domain-specific knowledge graphs.We show that this boosts performance in our low-resource scenario considerably.Further, we investigate the benefits of tabular structure as inductive bias compared to tables as linearized sequences.Our experiments confirm that a table transformer outperforms other baselines and that its tabular inductive bias is vital for convergence of transformer-based models.
Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder

Dense retrievers encode queries and documents and map them in an embedding space using pre-trained language models.These embeddings need to be high-dimensional to fit training signals and guarantee the retrieval effectiveness of dense retrievers.However, these highdimensional embeddings lead to larger index storage and higher retrieval latency.To reduce the embedding dimensions of dense retrieval, this paper proposes a Conditional Autoencoder (ConAE) to compress the high-dimensional embeddings to maintain the same embedding distribution and better recover the ranking features.Our experiments show that ConAE is effective in compressing embeddings by achieving comparable ranking performance with its teacher model and making the retrieval system more efficient.Our further analyses show that ConAE can alleviate the redundancy of the embeddings of dense retrieval with only one linear layer.All codes of this work are available at https://github.com/NEUIR/ConAE.
ROSE: Robust Selective Fine-tuning for Pre-trained Language Models

Even though the large-scale language models have achieved excellent performances, they suffer from various adversarial attacks.A large body of defense methods has been proposed.However, they are still limited due to redundant attack search spaces and the inability to defend against various types of attacks.In this work, we present a novel fine-tuning approach called RObust SEletive fine-tuning (ROSE) to address this issue.ROSE conducts selective updates when adapting pre-trained models to downstream tasks, filtering out invaluable and unrobust updates of parameters.Specifically, we propose two strategies: the first-order and second-order ROSE for selecting target robust parameters.The experimental results show that ROSE achieves significant improvements in adversarial robustness on various downstream NLP tasks, and the ensemble method even surpasses both variants above.Furthermore, ROSE can be easily incorporated into existing fine-tuning methods to improve their adversarial robustness further.The empirical analysis confirms that ROSE eliminates unrobust spurious updates during fine-tuning, leading to solutions corresponding to flatter and wider optima than the conventional method.Code is available at https: //github.com/jiangllan/ROSE.
Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning

Multiple pre-training objectives fill the vacancy of the understanding capability of single-objective language modeling, which serves the ultimate purpose of pre-trained language models (PrLMs), generalizing well on a mass of scenarios.However, learning multiple training objectives in a single model is challenging due to the unknown relative significance as well as the potential contrariety between them.Empirical studies have shown that the current objective sampling in an adhoc manual setting makes the learned language representation barely converge to the desired optimum.Thus, we propose MOMETAS, a novel adaptive sampler based on metalearning, which learns the latent sampling pattern on arbitrary pre-training objectives.Such a design is lightweight with negligible additional training overhead.To validate our approach, we adopt five objectives and conduct continual pre-training with BERT-base and BERT-large models, where MOMETAS demonstrates universal performance gain over other rule-based sampling strategies on 14 natural language processing tasks 1 .
Learning Label Modular Prompts for Text Classification in the Wild

Machine learning models usually assume i.i.d data during training and testing, but data and tasks in real world often change over time.To emulate the transient nature of real world, we propose a challenging but practical task: text classification in-the-wild, which introduces different non-stationary training/testing stages.Decomposing a complex task into modular components can enable robust generalisation under such non-stationary environment.However, current modular approaches in NLP do not take advantage of recent advances in parameter efficient tuning of pretrained language models.To close this gap, we propose MOD-ULARPROMPT, a label-modular prompt tuning framework for text classification tasks.In MOD-ULARPROMPT, the input prompt consists of a sequence of soft label prompts, each encoding modular knowledge related to the corresponding class label.In two of most formidable settings, MODULARPROMPT outperforms relevant baselines by a large margin demonstrating strong generalisation ability.We also conduct comprehensive analysis to validate whether the learned prompts satisfy properties of a modular representation. 1
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing

Despite their strong performance on many tasks, pre-trained language models have been shown to struggle on out-of-distribution compositional generalization.Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling.Can scaling up model size also improve compositional generalization in semantic parsing?We evaluate encoder-decoder models up to 11B parameters and decoder-only models up to 540B parameters, and compare model scaling curves for three different methods for applying a pre-trained language model to a new task: fine-tuning all parameters, prompt tuning, and in-context learning.We observe that fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional generalization in semantic parsing evaluations.In-context learning has positive scaling curves, but is generally outperformed by much smaller fine-tuned models.Prompt-tuning can outperform fine-tuning, suggesting further potential improvements from scaling as it exhibits a more positive scaling curve.Additionally, we identify several error trends that vary with model scale.For example, larger models are generally better at modeling the syntax of the output space, but are also more prone to certain types of overfitting.Overall, our study highlights limitations of current techniques for effectively leveraging model scale for compositional generalization, while our analysis also suggests promising directions for future work.
Alleviating Sparsity of Open Knowledge Graphs with Ternary Contrastive Learning

Sparsity of formal knowledge and roughness of non-ontological construction make sparsity problem particularly prominent in Open Knowledge Graphs (OpenKGs).Due to sparse links, learning effective representation for few-shot entities becomes difficult.We hypothesize that by introducing negative samples, a contrastive learning (CL) formulation could be beneficial in such scenarios.However, existing CL methods model KG triplets as binary objects of entities ignoring the relation-guided ternary propagation patterns and they are too generic, i.e., they ignore zero-shot, few-shot and synonymity problems that appear in OpenKGs.To address this, we propose TernaryCL, a CL framework based on ternary propagation patterns among head, relation and tail.TernaryCL designs Contrastive Entity and Contrastive Relation to mine ternary discriminative features with both negative entities and relations, introduces Contrastive Self to help zero-and few-shot entities learn discriminative features, Contrastive Synonym to model synonymous entities, and Contrastive Fusion to aggregate graph features from multiple paths.Extensive experiments on benchmarks demonstrate the superiority of TernaryCL over state-of-the-art models.
DANLI: Deliberative Agent for Following Natural Language Instructions

Recent years have seen an increasing amount of work on embodied AI agents that can perform tasks by following human language instructions.However, most of these agents are reactive, meaning that they simply learn and imitate behaviors encountered in the training data.These reactive agents are insufficient for longhorizon complex tasks.To address this limitation, we propose a neuro-symbolic deliberative agent that, while following language instructions, proactively applies reasoning and planning based on its neural and symbolic representations acquired from past experience (e.g., natural language and egocentric vision).We show that our deliberative agent achieves greater than 70% improvement over reactive baselines on the challenging TEACh benchmark.Moreover, the underlying reasoning and planning processes, together with our modular framework, offer impressive transparency and explainability to the behaviors of the agent.This enables an in-depth understanding of the agent's capabilities, which shed light on challenges and opportunities for future embodied agents for instruction following.The code is available at https://github.com/sled-group/DANLI.
Attack on Unfair ToS Clause Detection: A Case Study using Universal Adversarial Triggers

Recent work has demonstrated that natural language processing techniques can support consumer protection by automatically detecting unfair clauses in the Terms of Service (ToS) Agreement.This work demonstrates that transformer-based ToS analysis systems are vulnerable to adversarial attacks.We conduct experiments attacking an unfair-clause detector with universal adversarial triggers.Experiments show that a minor perturbation of the text can considerably reduce the detection performance.Moreover, to measure the detectability of the triggers, we conduct a detailed human evaluation study by collecting both answer accuracy and response time from the participants.The results show that the naturalness of the triggers remains key to tricking readers.
SMASH: Improving SMAll Language Models’ Few-SHot Ability with Prompt-Based Distillation

Large-scale language models coupled with prompts have shown remarkable performance on few-shot learning.However, through systematic experiments, we find that the few-shot performance of small language models is poor, and using prompts on them brings fewer improvements than on larger ones.In this paper, we propose SMASH, an approach to improve SMAll language models' few-SHot ability by training on intermediate tasks before promptbased fine-tuning on downstream tasks.We design intermediate tasks for sentence-pair tasks and sentiment classification tasks by creating training examples with prompt templates similar to downstream tasks using sentences sampled from a large-scale unsupervised corpus, and apply knowledge distillation to distill from outputs of larger pre-trained models as the training objective.We conduct extensive experiments and show that SMASH can make a 6layer DistilRoBRETa-base achieve comparable performance on few-shot datasets with a 12layer RoBERTa-base at a low cost.