Reviewerly/exHarmony huggingface dataset:
A Subspace-Based Analysis of Structured and Unstructured Representations in Image-Text Retrieval

In this paper, we specifically look at the image-text retrieval problem.Recent multimodal frameworks have shown that structured inputs and fine-tuning lead to consistent performance improvement.However, this paradigm has been challenged recently with newer Transformer-based models that can reach zero-shot state-of-the-art results despite not explicitly using structured data during pretraining.Since such strategies lead to increased computational resources, we seek to better understand their role in image-text retrieval by analyzing visual and text representations extracted with three multimodal frameworks: SGM, UNITER, and CLIP.To perform such analysis, we represent a single image or text as low-dimensional linear subspaces and perform retrieval based on subspace similarity.We chose this representation as subspaces give us the flexibility to model an entity based on feature sets, allowing us to observe how integrating or reducing information changes the representation of each entity.We analyze the performance of the selected models' features on two standard benchmark datasets.Our results indicate that heavily pre-training models can already lead to features with critical information representing each entity, with zeroshot UNITER features performing consistently better than fine-tuned features.Furthermore, while models can benefit from structured inputs, learning representations for objects and relationships separately, such as in SGM, likely causes a loss of crucial contextual information needed to obtain a compact cluster that can effectively represent a single entity.
Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations

Large pretrained multilingual language models (ML-LMs) have shown remarkable capabilities of zero-shot cross-lingual transfer, without direct cross-lingual supervision.While these results are promising, follow-up works found that, within the multilingual embedding spaces, there exists strong language identity information which hinders the expression of linguistic factors shared across languages.For semantic tasks like cross-lingual sentence retrieval, it is desired to remove such language identity signals to fully leverage semantic information.In this work, we provide a novel view of projecting away language-specific factors from a multilingual embedding space.Specifically, we discover that there exists a low-rank subspace that primarily encodes information irrelevant to semantics (e.g., syntactic information).To identify this subspace, we present a simple but effective unsupervised method based on singular value decomposition with multiple monolingual corpora as input.Once the subspace is found, we can directly project the original embeddings into the null space to boost language agnosticism without finetuning.We systematically evaluate our method on various tasks including the challenging language-agnostic QA retrieval task.Empirical results show that applying our method consistently leads to improvements over commonly used ML-LMs.
Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification

Social media is daily creating massive multimedia content with paired image and text, presenting the pressing need to automate the vision and language understanding for various multimodal classification tasks.Compared to the commonly researched visual-lingual data, social media posts tend to exhibit more implicit image-text relations.To better glue the crossmodal semantics therein, we capture hinting features from user comments, which are retrieved via jointly leveraging visual and lingual similarity.Afterwards, the classification tasks are explored via self-training in a teacherstudent framework, motivated by the usually limited labeled data scales in existing benchmarks.Substantial experiments are conducted on four multimodal social media benchmarks for image-text relation classification, sarcasm detection, sentiment classification, and hate speech detection.The results show that our method further advances the performance of previous state-of-the-art models, which do not employ comment modeling or self-training.
DynaMaR: Dynamic Prompt with Mask Token Representation

Recent research has shown that large language models pretrained using unsupervised approaches can achieve significant performance improvement on many downstream tasks.Typically when adapting these language models to downstream tasks, like a classification or regression task, we employ a finetuning paradigm in which the sentence representation from the language model is input to a task-specific head; the model is then fine-tuned end-to-end.However, with the emergence of models like GPT-3, prompt-based fine-tuning has been proven to be a successful approach for few-shot tasks.Inspired by this work, we study discrete prompt technologies in practice.There are two issues that arise with the standard prompt approach.First, it can overfit on the prompt template.Second, it requires manual effort to formulate the downstream task as a language model problem.In this paper, we propose an improvement to prompt-based finetuning that addresses these two issues.We refer to our approach as DynaMaR -Dynamic Prompt with Mask Token Representation.Results show that DynaMaR can achieve an average improvement of 10% in few-shot settings and improvement of 3.7% in data-rich settings over the standard fine-tuning approach on four e-commerce applications.
Parsing as Deduction Revisited: Using an Automatic Theorem Prover to Solve an SMT Model of a Minimalist Parser

We introduce a constraint-based parser for Minimalist Grammars (MG), implemented as a working computer program, that falls within the long established "Parsing as Deduction" framework.The parser takes as input an MG lexicon and a (partially specified) pairing of sound with meaning -i.e. a word sequence paired with a semantic representation -and, using an axiomatized logic, declaratively deduces syntactic derivations (i.e.parse trees) that comport with the specified interface conditions.The parser is built on the first axiomatization of MGs to use Satisfiability Modulo Theories (SMT), encoding in a constraint-based way the principles of minimalist syntax.The parser operates via a novel solution method: it assembles an SMT model of an MG derivation, translates the inputs into SMT formulae that constrain the model, and then solves the model using the Z3 SMT-solver, a high-performance automatic theorem prover; as the SMT-model has finite size (being bounded by the inputs), it is decidable and thus solvable in finite time.The output derivation is then recovered from the model solution.To demonstrate this, we run the parser on several representative inputs and examine how the output derivations differ when the inputs are partially vs. fully specified.We conclude by discussing the parser's extensibility and how a linguist can use it to automatically identify: (i) dependencies between input interface conditions and principles of syntax, and (ii) contradictions or redundancies between the model axioms encoding principles of syntax.
That’s so cute!: The CARE Dataset for Affective Response Detection

Social media plays an increasing role in our communication with friends and family, and in our consumption of entertainment and information. Hence, to design effective ranking functions for posts on social media, it would be useful to predict the affective responses of a post (e.g., whether it is likely to elicit feelings of entertainment, inspiration, or anger). Similar to work on emotion detection (which focuses on the affect of the publisher of the post), the traditional approach to recognizing affective response would involve an expensive investment in human annotation of training data. We create and publicly release CARE DB, a dataset of 230k social media post annotations according to seven affective responses using the Common Affective Response Expression (CARE) method. The CARE method is a means of leveraging the signal that is present in comments that are posted in response to a post, providing high-precision evidence about the affective response to the post without human annotation. Unlike human annotation, the annotation process we describe here can be iterated upon to expand the coverage of the method, particularly for new affective responses. We present experiments that demonstrate that the CARE annotations compare favorably with crowdsourced annotations. Finally, we use CARE DB to train competitive BERT-based models for predicting affective response as well as emotion detection, demonstrating the utility of the dataset for related tasks.
Efficiently Tuned Parameters Are Task Embeddings

Intermediate-task transfer can benefit a wide range of NLP tasks with properly selected source datasets.However, it is computationally infeasible to experiment with all intermediate transfer combinations, making choosing a useful source task a challenging problem.In this paper, we anticipate that task-specific parameters updated in parameter-efficient tuning methods are likely to encode task-specific information.Therefore, such parameters can be predictive for inter-task transferability.Thus, we propose to exploit these efficiently tuned parameters as off-the-shelf task embeddings for the efficient selection of source datasets for intermediate-task transfer.We experiment with 11 text classification tasks and 11 question answering tasks.Experimental results show that our approach can consistently outperform existing inter-task transferability prediction methods while being conceptually simple and computationally efficient.Our analysis also reveals that the ability of efficiently tuned parameters on transferability prediction is disentangled with their in-task performance.This allows us to use parameters from early checkpoints as task embeddings to further improve efficiency.
Cross-Encoder Data Annotation for Bi-Encoder Based Product Matching

Matching a seller listed item to an appropriate product is an important step for an e-commerce platform.With the recent advancement in deep learning, there are different encoder based approaches being proposed as solution.When textual data for two products are available, cross-encoder approaches encode them jointly while bi-encoder approaches encode them separately.Since cross-encoders are computationally heavy, approaches based on bi-encoders are a common practice for this challenge.In this paper, we propose cross-encoder data annotation; a technique to annotate or refine human annotated training data for bi-encoder models using a cross-encoder model.This technique enables us to build a robust model without annotation on newly collected training data or further improve model performance on annotated training data.We evaluate the cross-encoder data annotation on the product matching task using a real-world e-commerce dataset containing 104 million products.Experimental results show that the cross-encoder data annotation improves 4% absolute accuracy when no annotation for training data is available, and 2% absolute accuracy when annotation for training data is available.
Language Model Decomposition: Quantifying the Dependency and Correlation of Language Models

Pre-trained language models (LMs), such as BERT (Devlin et al., 2018) and its variants, have led to significant improvements on various NLP tasks in past years.However, a theoretical framework for studying their relationships is still missing.In this paper, we fill this gap by investigating the linear dependency between pre-trained LMs.The linear dependency of LMs is defined analogously to the linear dependency of vectors.We propose Language Model Decomposition (LMD) to represent a LM using a linear combination of other LMs as basis, and derive the closed-form solution.A goodness-of-fit metric for LMD similar to the coefficient of determination is defined and used to measure the linear dependency of a set of LMs.In experiments, we find that BERT and eleven (11) BERT-like LMs are 91% linearly dependent.This observation suggests that current state-of-the-art (SOTA) LMs are highly "correlated".To further advance SOTA we need more diverse and novel LMs that are less dependent on existing LMs.
Enhancing the Transformer Decoder with Transition-based Syntax

Notwithstanding recent advances, syntactic generalization remains a challenge for text decoders.While some studies showed gains from incorporating source-side symbolic syntactic and semantic structure into text generation Transformers, very little work addressed the decoding of such structure.We propose a general approach for tree decoding using a transition-based approach.Examining the challenging test case of incorporating Universal Dependencies syntax into machine translation, we present substantial improvements on test sets that focus on syntactic generalization, while presenting improved or comparable performance on standard MT benchmarks.Further qualitative analysis addresses cases where syntactic generalization in the vanilla Transformer decoder is inadequate and demonstrates the advantages afforded by integrating syntactic information. 1
Sentence Ambiguity, Grammaticality and Complexity Probes

It is unclear whether, how and where large pre-trained language models capture subtle linguistic traits like ambiguity, grammaticality and sentence complexity.We present results of automatic classification of these traits and compare their viability and patterns across representation types.We demonstrate that template-based datasets with surface-level artifacts should not be used for probing, careful comparisons with baselines should be done and that t-SNE plots should not be used to determine the presence of a feature among dense vectors representations.We also show how features might be highly localized in the layers for these models and get lost in the upper layers.
Towards Need-Based Spoken Language Understanding Model Updates: What Have We Learned?

In productionized machine learning systems, online model performance is known to deteriorate over time when there is a distributional drift between offline training and online application data. As a remedy, models are typically retrained at fixed time intervals, implying high computational and manual costs. This work aims at decreasing such costs in productionized, large-scale Spoken Language Understanding systems. In particular, we develop a need-based re-training strategy guided by an efficient drift detector and discuss the arising challenges including system complexity, overlapping model releases, observation limitation and the absence of annotated resources at runtime. We present empirical results on historical data and confirm the utility of our design decisions via an online A/B experiment.
Generating Classical Arabic Poetry using Pre-trained Models

Poetry generation tends to be a complicated task given meter and rhyme constraints.Previous work resorted to exhaustive methods inorder to employ poetic elements.In this paper we leave pre-trained models, GPT-J and BERTShared to recognize patterns of meters and rhyme to generate classical Arabic poetry and present our findings and results on how well both models could pick up on these classical Arabic poetic elements.
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer

Systems for knowledge-intensive tasks such as open-domain question answering (QA) usually consist of two stages: efficient retrieval of relevant documents from a large corpus and detailed reading of the selected documents to generate answers.Retrievers and readers are usually modeled separately, which necessitates a cumbersome implementation and is hard to train and adapt in an end-to-end fashion.In this paper, we revisit this design and eschew the separate architecture and training in favor of a single Transformer that performs Retrieval as Attention (ReAtt), and end-to-end training solely based on supervision from the end QA task.We demonstrate for the first time that a single model trained end-to-end can achieve both competitive retrieval and QA performance, matching or slightly outperforming state-of-the-art separately trained retrievers and readers.Moreover, end-to-end adaptation significantly boosts its performance on out-of-domain datasets in both supervised and unsupervised settings, making our model a simple and adaptable solution for knowledgeintensive tasks.Code and models are available at https://github.com/jzbjyb/ReAtt.
Fact Checking Machine Generated Text with Dependency Trees

Factual and logical errors made by Natural Language Generation (NLG) systems limit their applicability in many settings.We study this problem in a conversational search and recommendation setting, and observe that we can often make two simplifying assumptions in this domain: (i) there exists a body of structured knowledge we can use for verifying factuality of generated text; and (ii) the text to be factually assessed typically has a well-defined structure and style.Grounded in these assumptions, we propose a fast, unsupervised and explainable technique, DepChecker, that assesses factuality of input text based on rules derived from structured knowledge patterns and dependency relations with respect to the input text.We show that DepChecker outperforms state-of-the-art, general purpose fact-checking techniques in this special, but important case.
An Anchor-based Relative Position Embedding Method for Cross-Modal Tasks

Position Embedding (PE) is essential for transformer to capture the sequence ordering of input tokens.Despite its general effectiveness verified in Natural Language Processing (NLP) and Computer Vision (CV), its application in cross-modal tasks remains unexplored and suffers from two challenges: 1) the input text tokens and image patches are not aligned; 2) the encoding space of each modality is different, making it unavailable for feature comparison.In this paper, we propose a unified position embedding method for these problems, called AnChor-basEd Relative Position Embedding (ACE-RPE), in which we first introduce an anchor locating mechanism to bridge the semantic gap and locate anchors from different modalities.Then we conduct the distance calculation of each text token and image patch by computing their shortest paths from the located anchors.Last, we embed the anchor-based distance to guide the computation of crossattention.In this way, it calculates cross-modal relative position embeddings for cross-modal transformer.Benefiting from ACE-RPE, our method obtains new SOTA results on a wide range of benchmarks, such as Image-Text Retrieval on MS-COCO and Flickr30K, Visual Entailment on SNLI-VE, Visual Reasoning on NLVR2 and Weakly-supervised Visual Grounding on RefCOCO+.
Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representation

Chart-based models have shown great potential in unsupervised grammar induction, running recursively and hierarchically, but requiring O(n 3 ) time-complexity.The Recursive Transformer based on Differentiable Trees (R2D2) makes it possible to scale to large language model pretraining even with a complex tree encoder, by introducing a heuristic pruning method.However, its rule-based pruning process suffers from local optima and slow inference.In this paper, we propose a unified R2D2 method that overcomes these issues.We use a top-down unsupervised parser as a modelguided pruning method, which also enables parallel encoding during inference.Our parser casts parsing as a split point scoring task by first scoring all split points for a given sentence and then using the highest-scoring one to recursively split a span into two parts.The reverse order of the splits is considered as the order of pruning in the encoder.We optimize the unsupervised parser by minimizing the Kullback-Leibler distance between tree probabilities from the parser and the R2D2 model.Our experiments show that our Fast-R2D2 significantly improves the grammar induction quality and achieves competitive results in downstream tasks. 1
Authorship Verification for Arabic Short Texts Using Arabic Knowledge-Base Model (AraKB)

The issue of verifying authorship has been a controversial and much disputed subject within the field of digital forensics and cyber investigations.Although extensive research has been carried out on authorship verification tasks, few studies have analyzed Arabic social media texts.This paper seeks to overcome this limitation and presents a new knowledge-based model to enhance Natural Language Understanding and thereby improve authorship verification performance.The proposed model provided promising results that would benefit research for different Natural Language Processing tasks for Arabic.
SEEN: Structured Event Enhancement Network for Explainable Need Detection of Information Recall Assistance

When recalling life experiences, people often forget or confuse life events, which necessitates information recall services.Previous work on information recall focuses on providing such assistance reactively, i.e., by retrieving the life event of a given query.Proactively detecting the need for information recall services is rarely discussed.In this paper, we use a humanannotated life experience retelling dataset to detect the right time to trigger the information recall service.We propose a pilot modelstructured event enhancement network (SEEN) that detects life event inconsistency, additional information in life events, and forgotten events.A fusing mechanism is also proposed to incorporate event graphs of stories and enhance the textual representations.To explain the need detection results, SEEN simultaneously provides support evidence by selecting the related nodes from the event graph.Experimental results show that SEEN achieves promising performance in detecting information needs.In addition, the extracted evidence can be served as complementary information to remind users what events they may want to recall.➊ My brother decided to propose to his fiancée, Ellie.Afterward, ➋ She hosted the party in their backyard.It was so fun!They had a beautifully stocked bar with lots of alcohol.➌ The shots and cocktails were fantastic!PRE-RETOLD STORY POST-RETOLD STORY Match Match Conflict Forgotten My brother's engagement party was hosted in my hometown.➍ It was hosted by his fiancée, Ellen.➎ She had it outside in her backyard.Even though it was 90 degrees, we all had so much fun !➏ In addition, it was also the first time that my boyfriend met my family.
Rethinking Positional Encoding in Tree Transformer for Code Representation

Transformers are now widely used in code representation, and several recent works further develop tree Transformers to capture the syntactic structure in source code.Specifically, novel tree positional encodings have been proposed to incorporate inductive bias into Transformer.In this work, we propose a novel tree Transformer encoding node positions based on our new description method for tree structures.Technically, local and global soft bias shown in previous works is both introduced as positional encodings of our Transformer model.Our model finally outperforms strong baselines on code summarization and completion tasks across two languages, demonstrating our model's effectiveness.Besides, extensive experiments and ablation study shows that combining both local and global paradigms is still helpful in improving model performance.We release our code at https://github.com/AwdHanPeng/TreeTransformer.