Reviewerly/exHarmony huggingface dataset:
TableVLM: Multi-modal Pre-training for Table Structure Recognition

Tables are widely used in research and business, and are suitable for human consumption, but not easily machine-processable, particularly when tables are present in images.One of the main challenges to extracting data from images of tables is to accurately recognize table structures, especially for complex tables with cross rows and columns.In this study, we propose a novel multi-modal pre-training model for table structure recognition, named TableVLM.With a two-stream multi-modal transformer-based encoder-decoder architecture, TableVLM learns to capture rich table structure-related features by multiple carefullydesigned unsupervised objectives inspired by the notion of masked visual-language modeling.To pre-train this model, we also created a dataset, called ComplexTable, which consists of 1, 000K samples to be released publicly.Experiment results show that the model built on pre-trained TableVLM can improve the performance up to 1.97% in tree-editing-distancescore on ComplexTable.
Cross-Lingual Transfer with Target Language-Ready Task Adapters

Adapters have emerged as a modular and parameter-efficient approach to (zero-shot) cross-lingual transfer.The established MAD-X framework employs separate language and task adapters which can be arbitrarily combined to perform the transfer of any task to any target language.Subsequently, BAD-X, an extension of the MAD-X framework, achieves improved transfer at the cost of MAD-X's modularity by creating 'bilingual' adapters specific to the source-target language pair.In this work, we aim to take the best of both worlds by (i) fine-tuning task adapters adapted to the target language(s) (so-called 'target language-ready' (TLR) adapters) to maintain high transfer performance, but (ii) without sacrificing the highly modular design of MAD-X.The main idea of 'target language-ready' adapters is to resolve the training-vs-inference discrepancy of MAD-X: the task adapter 'sees' the target language adapter for the very first time during inference, and thus might not be fully compatible with it.We address this mismatch by exposing the task adapter to the target language adapter during training, and empirically validate several variants of the idea: in the simplest form, we alternate between using the source and target language adapters during task adapter training, which can be generalized to cycling over any set of language adapters.We evaluate different TLR-based transfer configurations with varying degrees of generality across a suite of standard cross-lingual benchmarks, and find that the most general (and thus most modular) configuration consistently outperforms MAD-X and BAD-X on most tasks and languages.
Should you marginalize over possible tokenizations?

Autoregressive language models (LMs) map token sequences to probabilities.The usual practice for computing the probability of any character string (e.g.English sentences) is to first transform it into a sequence of tokens that is scored by the model.However, there are exponentially many token sequences that represent any given string.To truly compute the probability of a string one should marginalize over all tokenizations, which is typically intractable.Here, we analyze whether the practice of ignoring the marginalization is justified.To this end, we devise an importance-sampling-based algorithm that allows us to compute estimates of the marginal probabilities and compare them to the default procedure in a range of state-ofthe-art models and datasets.Our results show that the gap in log-likelihood is no larger than 0.5% in most cases, but that it becomes more pronounced for data with long complex words.
LI-RAGE: Late Interaction Retrieval Augmented Generation with Explicit Signals for Open-Domain Table Question Answering

Recent open-domain TableQA models are typically implemented as retriever-reader pipelines.The retriever component is usually a variant of the Dense Passage Retriever, which computes the similarities between questions and tables based on a single representation of each.These fixed vectors can be insufficient to capture fine-grained features of potentially very big tables with heterogeneous row/column information.We address this limitation by 1) applying late interaction models which enforce a finer-grained interaction between question and table embeddings at retrieval time.In addition, we 2) incorporate a joint training scheme of the retriever and reader with explicit table-level signals, and 3) embed a binary relevance token as a prefix to the answer generated by the reader, so we can determine at inference time whether the table used to answer the question is reliable and filter accordingly.The combined strategies set a new state-to-the-art performance on two public open-domain TableQA datasets.
Efficient Document Embeddings via Self-Contrastive Bregman Divergence Learning

Learning quality document embeddings is a fundamental problem in natural language processing (NLP), information retrieval (IR), recommendation systems, and search engines.Despite recent advances in the development of transformer-based models that produce sentence embeddings with self-contrastive learning, the encoding of long documents (Ks of words) is still challenging with respect to both efficiency and quality considerations.Therefore, we train Longfomer-based document encoders using a state-of-the-art unsupervised contrastive learning method (SimCSE).Further on, we complement the baseline methodsiamese neural network-with additional convex neural networks based on functional Bregman divergence aiming to enhance the quality of the output document representations.We show that overall the combination of a self-contrastive siamese network and our proposed neural Bregman network outperforms the baselines in two linear classification settings on three long document topic classification tasks from the legal and biomedical domains.
Chemical Language Understanding Benchmark

In this paper, we introduce the benchmark 2 datasets named CLUB (Chemical 3 Language Understanding Benchmark) to 4 facilitate NLP research in the chemical 5 industry.We have 4 datasets consisted of 6 text and token classification tasks.As far as 7 we have recognized, it is one of the first 8 examples of chemical language 9 understanding benchmark datasets 10 consisted of tasks for both patent and 11 literature articles provided by industrial 12 organization.All the datasets are internally 13 made by chemists from scratch.Finally, we 14 evaluate the datasets on the various 15 language models based on BERT and 16 RoBERTa, and demonstrate the model 17 performs better when the domain of the pre-18 trained models are closer to chemistry 19 domain.We provide baselines for our 20 benchmark as 0.7818 in average, and we 21 hope this benchmark is used by many 22 researchers in both industry and academia.
Care4Lang at MEDIQA-Chat 2023: Fine-tuning Language Models for Classifying and Summarizing Clinical Dialogues

Summarizing medical conversations is one of the tasks proposed by MEDIQA-Chat to promote research on automatic clinical note generation from doctor-patient conversations.In this paper, we present our submission to this task using fine-tuned language models, including T5, BART and BioGPT models.The finetuned models are evaluated using ensemble metrics including ROUGE, BERTScore and BLEURT.Among the fine-tuned models, Flan-T5 achieved the highest aggregated score for dialogue summarization.
ARC-NLP at CASE 2022 Task 1: Ensemble Learning for Multilingual Protest Event Detection

Automated socio-political protest event detection is a challenging task when multiple languages are considered.In CASE 2022 Task 1, we propose ensemble learning methods for multilingual protest event detection in four subtasks with different granularity levels from documentlevel to entity-level.We develop an ensemble of fine-tuned Transformer-based language models, along with a post-processing step to regularize the predictions of our ensembles.Our approach places the first place in 6 out of 16 leaderboards organized in seven languages including English, Mandarin, and Turkish.
Conditional set generation using Seq2seq models

Conditional set generation learns a mapping from an input sequence of tokens to a set.Several NLP tasks, such as entity typing and dialogue emotion tagging, are instances of set generation.SEQ2SEQ models, a popular choice for set generation, treat a set as a sequence and do not fully leverage its key properties, namely order-invariance and cardinality.We propose a novel algorithm for effectively sampling informative orders over the combinatorial space of label orders.We jointly model the set cardinality and output by prepending the set size and taking advantage of the autoregressive factorization used by SEQ2SEQ models.Our method is a model-independent data augmentation approach that endows any SEQ2SEQ model with the signals of order-invariance and cardinality.Training a SEQ2SEQ model on this augmented data (without any additional annotations) gets an average relative improvement of 20% on four benchmark datasets across various models: BART-base, T5-11B, and GPT3-175B. 1
Characterizing Verbatim Short-Term Memory in Neural Language Models

When a language model is trained to predict natural language sequences, its prediction at each moment depends on a representation of prior context.What kind of information about the prior context can language models retrieve?We tested whether language models could retrieve the exact words that occurred previously in a text.In our paradigm, language models (transformers and an LSTM) processed English text in which a list of nouns occurred twice.We operationalized retrieval as the reduction in surprisal from the first to the second list.We found that the transformers retrieved both the identity and ordering of nouns from the first list.Further, the transformers' retrieval was markedly enhanced when they were trained on a larger corpus and with greater model depth.Lastly, their ability to index prior tokens was dependent on learned attention patterns.In contrast, the LSTM exhibited less precise retrieval, which was limited to list-initial tokens and to short intervening texts.The LSTM's retrieval was not sensitive to the order of nouns and it improved when the list was semantically coherent.We conclude that transformers implemented something akin to a working memory system that could flexibly retrieve individual token representations across arbitrary delays; conversely, the LSTM maintained a coarser and more rapidly-decaying semantic gist of prior tokens, weighted toward the earliest items.
PseudoReasoner: Leveraging Pseudo Labels for Commonsense Knowledge Base Population

Commonsense Knowledge Base (CSKB) Population aims at reasoning over unseen entities and assertions on CSKBs, and is an important yet hard commonsense reasoning task.One challenge is that it requires out-of-domain generalization ability as the source CSKB for training is of a relatively smaller scale (1M) while the whole candidate space for population is way larger (200M).We propose PseudoReasoner, a semi-supervised learning framework for CSKB population that uses a teacher model pre-trained on CSKBs to provide pseudo labels on the unlabeled candidate dataset for a student model to learn from.The teacher can be a generative model rather than restricted to discriminative models as previous works.In addition, we design a new filtering procedure for pseudo labels based on influence function and the student model's prediction to further improve the performance.The framework can improve the backbone model KG-BERT (RoBERTa-large) by 3.3 points on the overall performance and especially, 5.3 points on the out-of-domain performance, and achieves the state-of-the-art.Codes and data are available at https://github.com/HKUST-KnowComp/PseudoReasoner.
Word Representation Models for Arabic Dialect Identification

This paper describes the systems submitted by BFCAI team to Nuanced Arabic Dialect Identification (NADI) shared task 2022.Dialect identification task aims at detecting the source variant of a given text or speech segment automatically.There are two subtasks in NADI 2022, the first subtask for country-level identification and the second subtask for sentiment analysis.Our team participated in the first subtask.The proposed systems use Term Frequency Inverse/Document Frequency and word embeddings as vectorization models.Different machine learning algorithms have been used as classifiers.The proposed systems have been tested on two test sets: Test-A and Test-B.The proposed models achieved Macro-f1 score of 21.25% and 9.71% for Test-A and Test-B set respectively.On other hand, the best-performed submitted system achieved Macro-f1 score of 36.48% and 18.95% for Test-A and Test-B set respectively.
Jetsons at the FinNLP-2022 ERAI Task: BERT-Chinese for mining high MPP posts

In this paper, we discuss the various approaches by the Jetsons team for the "Pairwise Comparison" sub-task of the ERAI shared task to compare financial opinions for profitability and loss.Our BERT-Chinese model considers a pair of opinions and predicts the one with a higher maximum potential profit (MPP) with 62.07% accuracy.We analyze the performance of our approaches on both the MPP and maximal loss (ML) problems and deeply dive into why BERT-Chinese outperforms other models.
Improving Iterative Text Revision by Learning Where to Edit from Other Revision Tasks

Iterative text revision improves text quality by fixing grammatical errors, rephrasing for better readability or contextual appropriateness, or reorganizing sentence structures throughout a document.Most recent research has focused on understanding and classifying different types of edits in the iterative revision process from human-written text instead of building accurate and robust systems for iterative text revision.In this work, we aim to build an end-to-end text revision system that can iteratively generate helpful edits by explicitly detecting editable spans (where-to-edit) with their corresponding edit intents and then instructing a revision model to revise the detected edit spans.Leveraging datasets from other related text editing NLP tasks, combined with the specification of editable spans, leads our system to more accurately model the process of iterative text refinement, as evidenced by empirical results and human evaluations.Our system significantly outperforms previous baselines on our text revision tasks and other standard text revision tasks, including grammatical error correction, text simplification, sentence fusion, and style transfer.Through extensive qualitative and quantitative analysis, we make vital connections between edit intentions and writing quality, and better computational modeling of iterative text revisions.
LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification

Weakly supervised text classification methods typically train a deep neural classifier based on pseudo-labels.The quality of pseudo-labels is crucial to final performance but they are inevitably noisy due to their heuristic nature, so selecting the correct ones has a huge potential for performance boost.One straightforward solution is to select samples based on the softmax probability scores in the neural classifier corresponding to their pseudo-labels.However, we show through our experiments that such solutions are ineffective and unstable due to the erroneously high-confidence predictions from poorly calibrated models.Recent studies on the memorization effects of deep neural models suggest that these models first memorize training samples with clean labels and then those with noisy labels.Inspired by this observation, we propose a novel pseudolabel selection method LOPS that takes learning order of samples into consideration.We hypothesize that the learning order reflects the probability of wrong annotation in terms of ranking, and therefore, propose to select the samples that are learnt earlier.LOPS can be viewed as a strong performance-boost plug-in to most existing weakly-supervised text classification methods, as confirmed in extensive experiments on four real-world datasets.
Breaking through Inequality of Information Acquisition among Social Classes: A Modest Effort on Measuring “Fun”

With the identification of the inequality encoded in information acquisition among social classes, we propose to leverage a powerful concept that has never been studied as a linguistic construct, "fun", to deconstruct the inequality.Inspired by theories in sociology, we draw connection between social class and information cocoon, through the lens of fun, and hypothesize the measurement of "how fun one's dominating social cocoon is" to be an indicator of the social class of an individual.Following this, we propose an NLP framework to combat the issue by measuring how fun one's information cocoon is, and empower individuals to emancipate from their trapped cocoons.We position our work to be a domain-agnostic framework that can be deployed in a lot of downstream cases, and is one that aims to deconstruct, as opposed to reinforcing, the traditional social structure of beneficiaries (Jin et al., 2021).
Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC

This paper mainly describes the dma submission to the TempoWiC task, which achieves a macro-F1 score of 77.05% and attains the first place in this task.We first explore the impact of different pre-trained language models.Then we adopt data cleaning, data augmentation, and adversarial training strategies to enhance the model generalization and robustness.For further improvement, we integrate POS information and word semantic representation using a Mixture-of-Experts (MoE) approach.The experimental results show that MoE can overcome the feature overuse issue and combine the context, POS, and word semantic features well.Additionally, we use a model ensemble method for the final prediction, which has been proven effective by many research works.
GammaE: Gamma Embeddings for Logical Queries on Knowledge Graphs

Embedding knowledge graphs (KGs) for multihop logical reasoning is a challenging problem due to massive and complicated structures in many KGs.Recently, many promising works projected entities and queries into a geometric space to efficiently find answers.However, it remains challenging to model the negation and union operator.The negation operator has no strict boundaries, which generates overlapped embeddings and leads to obtaining ambiguous answers.An additional limitation is that the union operator is non-closure, which undermines the model to handle a series of union operators.To address these problems, we propose a novel probabilistic embedding model, namely Gamma Embeddings (GammaE), for encoding entities and queries to answer different types of FOL queries on KGs.We utilize the linear property and strong boundary support of the Gamma distribution to capture more features of entities and queries, which dramatically reduces model uncertainty.Furthermore, Gam-maE implements the Gamma mixture method to design the closed union operator.The performance of GammaE is validated on three large logical query datasets.Experimental results show that GammaE significantly outperforms state-of-the-art models on public benchmarks.
Diving Deep into Modes of Fact Hallucinations in Dialogue Systems

Knowledge Graph(KG) grounded conversations often use large pre-trained models and usually suffer from fact hallucination.Frequently entities with no references in knowledge sources and conversation history are introduced into responses, thus hindering the flow of the conversation-existing work attempt to overcome this issue by tweaking the training procedure or using a multi-step refining method.However, minimal effort is put into constructing an entity-level hallucination detection system, which would provide fine-grained signals that control fallacious content while generating responses.As a first step to address this issue, we dive deep to identify various modes of hallucination in KG-grounded chatbots through human feedback analysis.Secondly, we propose a series of perturbation strategies to create a synthetic dataset named FADE (FActual Dialogue Hallucination DEtection Dataset) 1 .Finally, we conduct comprehensive data analyses and create multiple baseline models for hallucination detection to compare against human-verified data and already established benchmarks.
Few-shot initializing of Active Learner via Meta-Learning

Despite the important evolutions in few-shot and zero-shot learning techniques, domain specific applications still require expert knowledge and significant effort in annotating and labeling a large volume of unstructured textual data.To mitigate this problem, active learning and meta-learning attempt to reach a high performance with the least amount of labeled data.In this paper, we introduce a novel approach to combine both lines of work by initializing an active learner with meta-learned parameters obtained through meta-training on tasks similar to the target task during active learning.In this approach we use the pre-trained BERT as our text-encoder and meta-learn its parameters with LEOPARD, which extends the model-agnostic meta-learning method by generating task dependent softmax weights to enable learning across tasks with different number of classes.We demonstrate the effectiveness of our method by performing active learning on five natural language understanding tasks and six datasets with five different acquisition functions.We train two different meta-initializations and we use the pre-trained BERT base initialization as baseline.We observe that our approach performs better than the baseline at low budget, especially when closely related tasks were present during meta-learning.Moreover, our results show that better performance in the initial phase, i.e., with fewer labeled samples, leads to better performance when larger acquisition batches are used.We also perform an ablation study of the proposed method, showing that active learning with only the metalearned weights is beneficial and adding the meta-learned learning rates and generating the softmax have negative consequences for the performance.
Leveraging Affirmative Interpretations from Negation Improves Natural Language Understanding

Negation poses a challenge in many natural language understanding tasks.Inspired by the fact that understanding a negated statement often requires humans to infer affirmative interpretations, in this paper we show that doing so benefits models for three natural language understanding tasks.We present an automated procedure to collect pairs of sentences with negation and their affirmative interpretations, resulting in over 150,000 pairs.Experimental results show that leveraging these pairs helps (a) T5 generate affirmative interpretations from negations in a previous benchmark, and (b) a RoBERTa-based classifier solve the task of natural language inference.We also leverage our pairs to build a plug-and-play neural generator that given a negated statement generates an affirmative interpretation.Then, we incorporate the pretrained generator into a RoBERTa-based classifier for sentiment analysis and show that doing so improves the results.Crucially, our proposal does not require any manual effort.* Work was done prior to joining Amazon.English-Norwegian (en-no) parallel sentences: (en) There is no more than one Truth.(no) Og det finnes kun en Sannhet.