Reviewerly/exHarmony huggingface dataset:
Re-appraising the Schema Linking for Text-to-SQL

Most text-to-SQL models, even though based on the same grammar decoder 1 , generate the SQL structure first and then fill in the SQL slots with the correct schema items.This second step depends on schema linking: aligning the entity references in the question with the schema columns or tables.This is generally approached via Exact Match based Schema Linking (EMSL) within a neural networkbased schema linking module.EMSL has become standard in text-to-SQL: many stateof-the-art models employ EMSL, with performance dropping significantly when the EMSL component is removed.In this work, however, we show that EMSL reduces robustness, rendering models vulnerable to synonym substitution and typos.Instead of relying on EMSL to make up for deficiencies in question-schema encoding, we show that using a pre-trained language model as an encoder can improve performance without using EMSL, giving a more robust model.We also study the design choice of the schema linking module, finding that a suitable design benefits performance and interpretability.Finally, based on the above study of schema linking, we introduce the grammar linking to help model align grammar references in the question with the SQL keywords. 2
Exploring the Curious Case of Code Prompts

Recent work has shown that prompting language models with code-like representations of natural language leads to performance improvements on structured reasoning tasks.However, such tasks comprise only a small subset of all natural language tasks.In our work, we seek to answer whether or not code-prompting is the preferred way of interacting with language models in general.We compare code and text prompts across three popular GPT models (davinci, code-davinci-002, and text-davinci-002) on a broader selection of tasks (e.g., QA, sentiment, summarization) and find that with few exceptions, code prompts do not consistently outperform text prompts.Furthermore, we show that the style of code prompt has a large effect on performance for some but not all tasks and that fine-tuning on text instructions leads to better relative performance of code prompts.
Gloss-Free End-to-End Sign Language Translation

In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations.Although intermediate representation like gloss has been proven effective, gloss annotations are hard to acquire, especially in large quantities.This limits the domain coverage of translation datasets, thus handicapping real-world applications.To mitigate this problem, we design the Gloss-Free End-to-end sign language translation framework (GloFE).Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation.Common concepts are extracted from the text and used as a weak form of intermediate representation.The global embedding of these concepts is used as a query for cross-attention to find the corresponding information within the learned visual features.In a contrastive manner, we encourage the similarity of query results between samples containing such concepts and decrease those that do not.We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign. 1
Laziness Is a Virtue When It Comes to Compositionality in Neural Semantic Parsing

Maxwell Crouse, Pavan Kapanipathi, Subhajit Chaudhury, Tahira Naseem, Ramon Fernandez Astudillo, Achille Fokoue, Tim Klinger. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
KoRC: Knowledge Oriented Reading Comprehension Benchmark for Deep Text Understanding

Deep text understanding, which requires the connections between a given document and prior knowledge beyond its text, has been highlighted by many benchmarks in recent years.However, these benchmarks have encountered two major limitations.On the one hand, most of them require human annotation of knowledge, which leads to limited knowledge coverage.On the other hand, they usually use choices or spans in the texts as the answers, which results in narrow answer space.To overcome these limitations, we build a new challenging benchmark named KORC in this paper.Compared with previous benchmarks, KORC has two advantages, i.e., broad knowledge coverage and flexible answer format.Specifically, we utilize massive knowledge bases to guide annotators or large language models (LLMs) to construct knowledgable questions.Moreover, we use labels in knowledge bases rather than spans or choices as the final answers.We test state-of-the-art models on KoRC and the experimental results show that the strongest baseline only achieves 68.3% and 30.0%F1 measure in the in-distribution and out-of-distribution test set, respectively.These results indicate that deep text understanding is still an unsolved challenge.The benchmark dataset, leaderboard, and baseline methods are released in https://github.com/THU-KEG/KoRC.
SongRewriter: A Chinese Song Rewriting System with Controllable Content and Rhyme Scheme

Although lyrics generation has achieved significant progress in recent years, it has limited practical applications because the generated lyrics cannot be performed without composing compatible melodies.In this work, we bridge this practical gap by proposing a song rewriting system which rewrites the lyrics of an existing song such that the generated lyrics are compatible with the rhythm of the existing melody and thus singable.In particular, we propose SongRewriter, 1 a controllable Chinese lyrics generation and editing system which assists users without prior knowledge of melody composition.The system is trained by a randomized multi-level masking strategy which produces a unified model for generating entirely new lyrics or editing a few fragments.To improve the controllabiliy of the generation process, we further incorporate a keyword prompt to control the lexical choices of the content and propose novel decoding constraints and a vowel modeling task to enable flexible end and internal rhyme schemes.While prior rhyming metrics are mainly for rap lyrics, we propose three novel rhyming evaluation metrics for song lyrics.Both automatic and human evaluations show that the proposed model performs better than the state-of-the-art models in both contents and rhyming quality.
Sakura at SemEval-2023 Task 2: Data Augmentation via Translation

We demonstrate a simple yet effective approach to augmenting training data for multilingual named entity recognition using machine translation.The named entity spans from the original sentences are transferred to the translations via word alignment and then filtered with the baseline recognizer to retain high quality annotations.The proposed data augmentation approach improves the baseline performance of XLM-Roberta on the multilingual dataset.
iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?

There are two competing approaches for modelling annotator disagreement: distributional soft-labelling approaches (which aim to capture the level of disagreement) or modelling perspectives of individual annotators or groups thereof.We adapt a multi-task architecturewhich has previously shown success in modelling perspectives-to evaluate its performance on the SEMEVAL Task 11.We do so by combining both approaches, i.e. predicting individual annotator perspectives as an interim step towards predicting annotator disagreement.Despite its previous success, we found that a multi-task approach performed poorly on datasets which contained distinct annotator opinions, suggesting that this approach may not always be suitable when modelling perspectives.Furthermore, our results explain that while strongly perspectivist approaches might not achieve state-of-the-art performance according to evaluation metrics used by distributional approaches, our approach allows for a more nuanced understanding of individual perspectives present in the data.We argue that perspectivist approaches are preferable because they enable decision makers to amplify minority views, and that it is important to re-evaluate metrics to reflect this goal.
oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes

In this paper, we introduce the range of oBERTa language models, an easy-to-use set of language models which allows Natural Language Processing (NLP) practitioners to obtain between 3.8 and 24.3 times faster models without expertise in model compression.Specifically, oBERTa extends existing work on pruning, knowledge distillation, and quantization and leverages frozen embeddings, improves distillation, and model initialization to deliver higher accuracy on a broad range of transfer tasks.In generating oBERTa, we explore how the highly optimized RoBERTa differs from the BERT for pruning during pre-training and finetuning.We find it less amenable to compression during fine-tuning.We explore the use of oBERTa on seven representative NLP tasks and find that the improved compression techniques allow a pruned oBERTa model to match the performance of BERT base and exceed the performance of Prune OFA Large on the SQUAD V1.1 Question Answering dataset, despite being 8x and 2x respectively faster in inference.We release our code, training regimes, and associated model for broad usage to encourage usage and experimentation. 1,2
Visually-Enhanced Phrase Understanding

Large-scale vision-language pre-training has exhibited strong performance in various visual and textual understanding tasks.Recently, the textual encoders of multi-modal pre-trained models have been shown to generate highquality textual representations, which often outperform models that are purely text-based, such as BERT.In this study, our objective is to utilize both textual and visual encoders of multimodal pre-trained models to enhance language understanding tasks.We achieve this by generating an image associated with a textual prompt, thus enriching the representation of a phrase for downstream tasks.Results from experiments conducted on four benchmark datasets demonstrate that our proposed method, which leverages visually-enhanced text representations, significantly improves performance in the entity clustering task.
Commonsense Knowledge Graph Completion Via Contrastive Pretraining and Node Clustering

The nodes in the commonsense knowledge graph (CSKG) are normally represented by free-form short text (e.g., word or phrase).Different nodes may represent the same concept.This leads to the problems of edge sparsity and node redundancy, which challenges CSKG representation and completion.On the one hand, edge sparsity limits the performance of graph representation learning; On the other hand, node redundancy makes different nodes corresponding to the same concept have inconsistent relations with other nodes.To address the two problems, we propose a new CSKG completion framework based on Contrastive Pretraining and Node Clustering (CPNC).Contrastive Pretraining constructs positive and negative head-tail node pairs on CSKG and utilizes contrastive learning to obtain better semantic node representation.Node Clustering aggregates nodes with the same concept into a latent concept, assisting the task of CSKG completion.We evaluate our CPNC approach on two CSKG completion benchmarks (CN-100K and ATOMIC), where CPNC outperforms the stateof-the-art methods.Extensive experiments demonstrate that both Contrastive Pretraining and Node Clustering can significantly improve the performance of CSKG completion.The source code of CPNC is publicly available on https://github.com/NUSTM/CPNC.
ACCEPT at SemEval-2023 Task 3: An Ensemble-based Approach to Multilingual Framing Detection

This paper describes the system and experimental results of an ensemble-based approach to multilingual framing detection for the submission of the ACCEPT team to the SemEval-2023 Task 3 on Framing Detection (Subtask 2).The approach is based on an ensemble that combines three different methods: a classifier based on large language models, a classifier based on static word embeddings, and an approach that uses external commonsense knowledge graphs, in particular, ConceptNet.The results of the three classification heads are aggregated into an overall prediction for each frame class.Our best submission yielded a micro F 1 -score of 50.69% (rank 10) and a macro F 1 -score of 50.20% (rank 3) for English articles.Our experimental results show that static word embeddings and knowledge graphs are useful components for frame detection, while the ensemble of all three methods combines the strengths of our three proposed methods.Through system ablations, we show that the commonsenseguided knowledge graphs are the outperforming method for many languages.
Revealing the Blind Spot of Sentence Encoder Evaluation by HEROS

Existing sentence textual similarity benchmark datasets only use a single number to summarize how similar the sentence encoder's decision is to humans'.However, it is unclear what kind of sentence pairs a sentence encoder (SE) would consider similar.Moreover, existing SE benchmarks mainly consider sentence pairs with low lexical overlap, so it is unclear how the SEs behave when two sentences have high lexical overlap.We introduce a high-quality SE diagnostic dataset, HEROS.HEROS is constructed by transforming an original sentence into a new sentence based on certain rules to form a minimal pair, and the minimal pair has high lexical overlaps.The rules include replacing a word with a synonym, an antonym, a typo, a random word, and converting the original sentence into its negation.Different rules yield different subsets of HEROS.By systematically comparing the performance of over 60 supervised and unsupervised SEs on HEROS, we reveal that most unsupervised sentence encoders are insensitive to negation.We find the datasets used to train the SE are the main determinants of what kind of sentence pairs an SE considers similar.We also show that even if two SEs have similar performance on STS benchmarks, they can have very different behavior on HEROS.Our result reveals the blind spot of traditional STS benchmarks when evaluating SEs. 1
Disfluency Generation for More Robust Dialogue Systems

Disfluencies in user utterances can trigger a chain of errors impacting all the modules of a dialogue system: natural language understanding, dialogue state tracking, and response generation.In this work, we first analyze existing dialogue datasets commonly used in research and show that they only contain a marginal number of disfluent utterances.Due to this relative absence of disfluencies in their training data, dialogue systems may then critically fail when exposed to disfluent utterances.Following this observation, we propose to augment existing datasets with disfluent user utterances by paraphrasing fluent utterances into disfluent ones.Relying on a pre-trained language model, our few-shot disfluent paraphraser guided by a disfluency classifier can generate useful disfluent utterances for training better dialogue systems.We report on improvements for both dialogue state tracking and response generation when the dialogue systems are trained on datasets augmented with our disfluent utterances.
CoMix: Guide Transformers to Code-Mix using POS structure and Phonetics

Code-mixing is ubiquitous in multilingual societies, which makes it vital to build models for code-mixed data to power human language interfaces. Existing multilingual transformer models trained on pure corpora lack the ability to intermix words of one language into the structure of another. These models are also not robust to orthographic variations. We propose CoMixCoMix is not a trademark and only used to refer to our models for code-mixed data for presentational brevity., a pretraining approach to improve representation of code-mixed data in transformer models by incorporating phonetic signals, a modified attention mechanism, and weak supervision guided generation by parts-of-speech constraints. We show that CoMix improves performance across four code-mixed tasks: machine translation, sequence classification, named entity recognition (NER), and abstractive summarization. It also achieves the new SOTA performance for English-Hinglish translation and NER on LINCE Leaderboard and provides better generalization on out-of-domain translation. Motivated by variations in human annotations, we also propose a new family of metrics based on phonetics and demonstrate that the phonetic variant of BLEU correlates better with human judgement than BLEU on code-mixed text.
“Let’s not Quote out of Context”: Unified Vision-Language Pretraining for Context Assisted Image Captioning

Well-formed context aware image captions and tags in enterprise content such as marketing material are critical to ensure their brand presence and content recall.Manual creation and updates to ensure the same is non trivial given the scale and the tedium towards this task.We propose a new unified Vision-Language (VL) model based on the One For All (OFA) model, with a focus on context-assisted image captioning where the caption is generated based on both the image and its context.Our approach aims to overcome the context-independent (image and text are treated independently) nature of the existing approaches.We exploit context by pretraining our model with datasets of three tasks-news image captioning where the news article is the context, contextual visual entailment, and keyword extraction from the context.The second pretraining task is a new VL task, and we construct and release two datasets for the task with 1.1M and 2.2K data instances.Our system achieves state-of-theart results with an improvement of up to 8.34 CIDEr score on the benchmark news image captioning datasets.To the best of our knowledge, ours is the first effort at incorporating contextual information in pretraining the models for the VL tasks.
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning

We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion.It provides the benefits of multitask learning but leverages distributed computation with limited communication and eliminates the need for shared data.Consequentially, ColD Fusion can give rise to a synergistic loop, where finetuned models can be recycled to continually improve the pretrained model they are based upon.We show that ColD Fusion yields comparable benefits to multitask training by producing a model that (a) attains strong performance on all of the datasets it was trained on; and (b) is a better starting point for finetuning on unseen datasets.We show that ColD Fusion outperforms RoBERTa and even previous multitask models.Specifically, when training and testing on 35 diverse datasets, ColD Fusion-based model outperforms RoBERTa by 2.33 points on average without any changes to the architecture. 1
Towards Diverse and Effective Question-Answer Pair Generation from Children Storybooks

Recent advances in QA pair generation (QAG) have raised interest in applying this technique to the educational field.However, the diversity of QA types remains a challenge despite its contributions to comprehensive learning and assessment of children.In this paper, we propose a QAG framework that enhances QA type diversity by producing different interrogative sentences and implicit/explicit answers.Our framework comprises a QFS-based answer generator, an iterative QA generator, and a relevancy-aware ranker.The two generators aim to expand the number of candidates while covering various types.The ranker trained on the in-context negative samples clarifies the top-N outputs based on the ranking score.Extensive evaluations and detailed analyses demonstrate that our approach outperforms previous state-of-the-art results by significant margins, achieving improved diversity and quality.Our task-oriented processes are consistent with real-world demand, which highlights our system's high applicability.
TechSSN at SemEval-2023 Task 12: Monolingual Sentiment Classification in Hausa Tweets

This paper elaborates on our work in designing a system for SemEval 2023 Task 12: AfriSentiSemEval, which involves sentiment analysis for low-resource African languages using the Twitter dataset. We utilised a pre-trained model to perform sentiment classification in Hausa language tweets. We used a multilingual version of the roBERTa model, which is pretrained on 100 languages, to classify sentiments in Hausa. To tokenize the text, we used the AfriBERTa model, which is specifically pretrained on African languages.
MetaFill: Text Infilling for Meta-Path Generation on Heterogeneous Information Networks

Heterogeneous Information Network (HIN) is essential to study complicated networks containing multiple edge types and node types.Meta-path, a sequence of node types and edge types, is the core technique to embed HINs.Since manually curating meta-paths is timeconsuming, there is a pressing need to develop automated meta-path generation approaches.Existing meta-path generation approaches cannot fully exploit the rich textual information in HINs, such as node names and edge type names.To address this problem, we propose MetaFill, a text-infilling-based approach for meta-path generation.The key idea of MetaFill is to formulate meta-path identification problem as a word sequence infilling problem, which can be advanced by Pretrained Language Models (PLMs).We observed the superior performance of MetaFill against existing meta-path generation methods and graph embedding methods that do not leverage meta-paths in both link prediction and node classification on two real-world HIN datasets.We further demonstrated how MetaFill can accurately classify edges in the zero-shot setting, where existing approaches cannot generate any meta-paths.MetaFill exploits PLMs to generate meta-paths for graph embedding, opening up new avenues for language model applications in graph analysis.
Making Pretrained Language Models Good Long-tailed Learners

Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge.This motivates us to check the hypothesis that prompt-tuning is also a promising choice for long-tailed classification, since the tail classes are intuitively few-shot ones.To achieve this aim, we conduct empirical studies to examine the hypothesis.The results demonstrate that prompt-tuning makes pretrained language models at least good long-tailed learners.For intuitions on why prompt-tuning can achieve good performance in long-tailed classification, we carry out in-depth analyses by progressively bridging the gap between prompttuning and commonly used finetuning.The summary is that the classifier structure and parameterization form the key to making good long-tailed learners, in comparison with the less important input structure.Finally, we verify the applicability of our finding to few-shot classification. 1
Sentence Representation Learning with Generative Objective rather than Contrastive Objective

Though offering amazing contextualized tokenlevel representations, current pre-trained language models take less attention on accurately acquiring sentence-level representation during their self-supervised pre-training.However, contrastive objectives which dominate the current sentence representation learning bring little linguistic interpretability and no performance guarantee on downstream semantic tasks.We instead propose a novel generative self-supervised learning objective based on phrase reconstruction.To overcome the drawbacks of previous generative methods, we carefully model intra-sentence structure by breaking down one sentence into pieces of important phrases.Empirical studies show that our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods not only on the STS benchmarks, but also on downstream semantic retrieval and reranking tasks.Our code is available at https://github.com/chengzhipanpan/PaSeR.