Reviewerly/exHarmony huggingface dataset:
Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning

Large Language Models (LLMs) are known to memorize significant portions of their training data.Parts of this memorized content have been shown to be extractable by simply querying the model, which poses a privacy risk.We present a novel approach which uses prompttuning to control the extraction rates of memorized content in LLMs.We present two prompt training strategies to increase and decrease extraction rates, which correspond to an attack and a defense, respectively.We demonstrate the effectiveness of our techniques by using models from the GPT-Neo family on a public benchmark.For the 1.3B parameter GPT-Neo model, our attack yields a 9.3 percentage point increase in extraction rate compared to our baseline.Our defense can be tuned to achieve different privacy-utility trade-offs by a user-specified hyperparameter.We achieve an extraction rate reduction of up to 97.7% relative to our baseline, with a perplexity increase of 16.9%.
From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding

Current state-of-the-art models for natural language understanding require a preprocessing step to convert raw text into discrete tokens.This process known as tokenization relies on a pre-built vocabulary of words or sub-word morphemes.This fixed vocabulary limits the model's robustness to spelling errors and its capacity to adapt to new domains.In this work, we introduce a novel open-vocabulary language model that adopts a hierarchical two-level approach: one at the word level and another at the sequence level.Concretely, we design an intraword module that uses a shallow Transformer architecture to learn word representations from their characters, and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence.Our model thus directly operates on character sequences with explicit awareness of word boundaries, but without biased sub-word or word-level vocabulary.Experiments on various downstream tasks show that our method outperforms strong baselines.We also demonstrate that our hierarchical model is robust to textual corruption and domain shift.
Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training

Prior work shows that it is possible to expand pretrained Masked Language Models (MLMs) to new languages by learning a new set of embeddings, while keeping the transformer body frozen.Despite learning a small subset of parameters, this approach is not computeefficient, as training the new embeddings requires a full forward and backward pass over the entire model.We propose mini-model adaptation, a compute-efficient alternative that builds a shallow mini-model from a fraction of a large model's parameters.New languagespecific embeddings can then be efficiently trained over the mini-model and plugged into the aligned large model for rapid cross-lingual transfer.We explore two approaches to learn mini-models: MINIJOINT, which jointly pretrains the primary model and the mini-model using a single transformer with a secondary MLM head at a middle layer; and MINIPOST, where we start from a regular pretrained model, build a mini-model by extracting and freezing a few layers, and learn a small number of parameters on top.Experiments on XNLI, MLQA and PAWS-X show that mini-model adaptation matches the performance of the standard approach using 2.3x less compute on average.
VTCC-NER at SemEval-2023 Task 6: An Ensemble Pre-trained Language Models for Named Entity Recognition

We propose an ensemble method that combines several pre-trained language models to enhance entity recognition in legal text.Our approach achieved a 90.9873%F1 score on the private test set, ranking 2nd on the leaderboard for Se-mEval 2023 Task 6, Subtask B -Legal Named Entities Extraction.Our code is available for further exploitation at: https://github.com/tqgminh/SemEval2023_LegalNER_VTCC.
Francis Wilde at SemEval-2023 Task 5: Clickbait Spoiler Type Identification with Transformers

Clickbait is the text or a thumbnail image that entices the user to click the accompanying link.Clickbait posts hide the critical elements of the article and reveal partial information in the title, which arouses sufficient curiosity and motivates the user to click the link.In this work, we identify the kind of spoiler given a clickbait title.We formulate this as a text classification problem.We finetune pretrained transformer models on the title of the post and build models for the clickbait-spoiler classification.We achieve a balanced accuracy of 0.70 which is close to the baseline.
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions

Despite the widespread success of Transformers on NLP tasks, recent works have found that they struggle to model several formal languages when compared to recurrent models.This raises the question of why Transformers perform well in practice and whether they have any properties that enable them to generalize better than recurrent models.In this work, we conduct an extensive empirical study on Boolean functions to demonstrate the following: (i) Random Transformers are relatively more biased towards functions of low sensitivity.(ii) When trained on Boolean functions, both Transformers and LSTMs prioritize learning functions of low sensitivity, with Transformers ultimately converging to functions of lower sensitivity.(iii) On sparse Boolean functions which have low sensitivity, we find that Transformers generalize near perfectly even in the presence of noisy labels whereas LSTMs overfit and achieve poor generalization accuracy.Overall, our results provide strong quantifiable evidence that suggests differences in the inductive biases of Transformers and recurrent models which may help explain Transformer's effective generalization performance despite relatively limited expressiveness.
FIT BUT at SemEval-2023 Task 12: Sentiment Without Borders - Multilingual Domain Adaptation for Low-Resource Sentiment Classification

This paper presents our proposed method for SemEval-2023 Task 12, which focuses on sentiment analysis for low-resource African languages.Our method utilizes a language-centric domain adaptation approach which is based on adversarial training, where a small version of Afro-XLM-Roberta serves as a generator model and a feed-forward network as a discriminator.We participated in all three subtasks: monolingual (12 tracks), multilingual (1 track), and zero-shot (2 tracks).Our results show an improvement in weighted F1 for 13 out of 15 tracks with a maximum increase of 4.3 points for Moroccan Arabic compared to the baseline.We observed that using language family-based labels along with sequence-level input representations for the discriminator model improves the quality of the cross-lingual sentiment analysis for the languages unseen during the training.Additionally, our experimental results suggest that training the system on languages that are close in a language families tree enhances the quality of sentiment analysis for low-resource languages.Lastly, the computational complexity of the prediction step was kept at the same level which makes the approach to be interesting from a practical perspective.The code of the approach can be found in our repository 1 .
Learning from Partially Annotated Data: Example-aware Creation of Gap-filling Exercises for Language Learning

Since performing exercises (including, e.g., practice tests) forms a crucial component of learning, and creating such exercises requires non-trivial effort from the teacher.There is a great value in automatic exercise generation in digital tools in education.In this paper, we particularly focus on automatic creation of gapfilling exercises for language learning, specifically grammar exercises.Since providing any annotation in this domain requires human expert effort, we aim to avoid it entirely and explore the task of converting existing texts into new gap-filling exercises, purely based on an example exercise, without explicit instruction or detailed annotation of the intended grammar topics.We contribute (i) a novel neural network architecture specifically designed for aforementioned gap-filling exercise generation task, and (ii) a real-world benchmark dataset for French grammar.We show that our model for this French grammar gap-filling exercise generation outperforms a competitive baseline classifier by 8% in F1 percentage points, achieving an average F1 score of 82%.Our model implementation and the dataset are made publicly available 1 to foster future research, thus offering a standardized evaluation and baseline solution of the proposed partially annotated data prediction task in grammar exercise creation.
CMU’s IWSLT 2023 Simultaneous Speech Translation System

This paper describes CMU's submission to the IWSLT 2023 simultaneous speech translation shared task for translating English speech to both German text and speech in a streaming fashion.We first build offline speech-totext (ST) models using the joint CTC/attention framework.These models also use WavLM front-end features and mBART decoder initialization.We adapt our offline ST models for simultaneous speech-to-text translation (SST) by 1) incrementally encoding chunks of input speech, re-computing encoder states for each new chunk and 2) incrementally decoding output text, pruning beam search hypotheses to 1best after processing each chunk.We then build text-to-speech (TTS) models using the VITS framework and achieve simultaneous speechto-speech translation (SS2ST) by cascading our SST and TTS models.
Sina at SemEval-2023 Task 4: A Class-Token Attention-based Model for Human Value Detection

The human values expressed in argumentative texts can provide valuable insights into the culture of a society.They can be helpful in various applications such as value-based profiling and ethical analysis.However, one of the first steps in achieving this goal is to detect the category of human value from an argument accurately.This task is challenging due to the lack of data and the need for philosophical inference.It also can be challenging for humans to classify arguments according to their underlying human values.This paper elaborates on our model for the SemEval 2023 Task 4 on human value detection.We propose a class-token attentionbased model and evaluate it against baseline models, including finetuned BERT language model and a keyword-based approach.
Measuring Your ASTE Models in The Wild: A Diversified Multi-domain Dataset For Aspect Sentiment Triplet Extraction

Aspect Sentiment Triplet Extraction (ASTE) is widely used in various applications.However, existing ASTE datasets are limited in their ability to represent real-world scenarios, hindering the advancement of research in this area.In this paper, we introduce a new dataset, named DMASTE, which is manually annotated to better fit real-world scenarios by providing more diverse and realistic reviews for the task.The dataset includes various lengths, diverse expressions, more aspect types, and more domains than existing datasets.We conduct extensive experiments on DMASTE in multiple settings to evaluate previous ASTE approaches.Empirical results demonstrate that DMASTE is a more challenging ASTE dataset.Further analyses of in-domain and cross-domain settings provide promising directions for future research.
BLIND: Bias Removal With No Demographics

Models trained on real-world data tend to imitate and amplify social biases.Common methods to mitigate biases require prior information on the types of biases that should be mitigated (e.g., gender or racial bias) and the social groups associated with each data sample.In this work, we introduce BLIND, a method for bias removal with no prior knowledge of the demographics in the dataset.While training a model on a downstream task, BLIND detects biased samples using an auxiliary model that predicts the main model's success, and downweights those samples during the training process.Experiments with racial and gender biases in sentiment classification and occupation classification tasks demonstrate that BLIND mitigates social biases without relying on a costly demographic annotation process.Our method is competitive with other methods that require demographic information and sometimes even surpasses them. 1
Neural Networks Against (and For) Self-Training: Classification with Small Labeled and Large Unlabeled Sets

We propose a semi-supervised text classifier based on self-training using one positive and one negative property of neural networks.One of the weaknesses of self-training is the semantic drift problem, where noisy pseudo-labels accumulate over iterations and consequently the error rate soars.In order to tackle this challenge, we reshape the role of pseudo-labels and create a hierarchical order of information.In addition, a crucial step in self-training is to use the classifier confidence prediction to select the best candidate pseudo-labels.This step cannot be efficiently done by neural networks, because it is known that their output is poorly calibrated.To overcome this challenge, we propose a hybrid metric to replace the plain confidence measurement.Our metric takes into account the prediction uncertainty via a subsampling technique.We evaluate our model in a set of five standard benchmarks, and show that it significantly outperforms a set of ten diverse baseline models.Furthermore, we show that the improvement achieved by our model is additive to language model pretraining, which is a widely used technique for using unlabeled documents.
Alexa at SemEval-2023 Task 10: Ensemble Modeling of DeBERTa and BERT Variations for Identifying Sexist Text

This study presents an ensemble approach for detecting sexist text in the context of the Semeval-2023 task 10.Our approach leverages 18 models, including DeBERTa-v3-base models with different input sequence lengths, a BERT-based model trained on identifying hate speech, and three more models pre-trained on the task's unlabeled data with varying input lengths.The results of our framework on the development set show an f1-score of 84.92% and on the testing set 84.55%, effectively demonstrating the strength of the ensemble approach in getting accurate results.
Randomized Positional Encodings Boost Length Generalization of Transformers

Transformers have impressive generalization capabilities on tasks with a fixed context length.However, they fail to generalize to sequences of arbitrary length, even for seemingly simple tasks such as duplicating a string.Moreover, simply training on longer sequences is inefficient due to the quadratic computation complexity of the global attention mechanism.In this work, we demonstrate that this failure mode is linked to positional encodings being out-of-distribution for longer sequences (even for relative encodings) and introduce a novel family of positional encodings that can overcome this problem.Concretely, our randomized positional encoding scheme simulates the positions of longer sequences and randomly selects an ordered subset to fit the sequence's length.Our large-scale empirical evaluation of 6000 models across 15 algorithmic reasoning tasks shows that our method allows Transformers to generalize to sequences of unseen length (increasing test accuracy by 12.0% on average).
IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition Using Knowledge Bases

Named Entity Recognition (NER) is a core natural language processing task in which pretrained language models have shown remarkable performance.However, standard benchmarks like CoNLL 2003 (Tjong Kim Sang andDe Meulder, 2003) do not address many of the challenges that deployed NER systems face, such as having to classify emerging or complex entities in a fine-grained way.In this paper we present a novel NER cascade approach comprising three steps: first, identifying candidate entities in the input sentence; second, linking the each candidate to an existing knowledge base; third, predicting the fine-grained category for each entity candidate.We empirically demonstrate the significance of external knowledge bases in accurately classifying finegrained and emerging entities.Our system exhibits robust performance in the MultiCoNER2 (Fetahu et al., 2023b) shared task, even in the low-resource language setting where we leverage knowledge bases of high-resource languages.
Trading Syntax Trees for Wordpieces: Target-oriented Opinion Words Extraction with Wordpieces and Aspect Enhancement

State-of-the-art target-oriented opinion word extraction (TOWE) models typically use BERTbased text encoders that operate on the word level, along with graph convolutional networks (GCNs) that incorporate syntactic information extracted from syntax trees.These methods achieve limited gains with GCNs and have difficulty using BERT wordpieces.Meanwhile, BERT wordpieces are known to be effective at representing rare words or words with insufficient context information.To address this issue, this work trades syntax trees for BERT wordpieces by entirely removing the GCN component from the methods' architectures.To enhance TOWE performance, we tackle the issue of aspect representation loss during encoding.Instead of solely utilizing a sentence as the input, we use a sentence-aspect pair.Our relatively simple approach achieves state-of-the-art results on benchmark datasets and should serve as a strong baseline for further research. Sentence:Such an awesome surfboard Wordpieces: 'such', 'an', 'awesome', 'surf', '##board' Sentence:A great snowboard which holds edges well when riding on snow.
I already said that! Degenerating redundant questions in open-domain dialogue systems.

Neural text generation models have achieved remarkable success in carrying on short opendomain conversations.However, their performance degrades significantly in the long term, especially in their ability to ask coherent questions.A significant issue is the generation of redundant questions where the answer has already been provided by the user.We adapt and evaluate different methods, including negative training, decoding, and classification, to mitigate the redundancy problem.We also propose a simple yet effective method for generating training data without the need for crowdsourcing human-human or human-bot conversations.Experiments with the BlenderBot model show that our combined method significantly reduces the rate of redundant questions from 27.2% to 8.7%, while improving the quality of the original model.The code, dataset, and trained models can be found at our repository 1 .
Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection

Stance Detection is concerned with identifying the attitudes expressed by an author towards a target of interest.This task spans a variety of domains ranging from social media opinion identification to detecting the stance for a legal claim.However, the framing of the task varies within these domains, in terms of the data collection protocol, the label dictionary and the number of available annotations.Furthermore, these stance annotations are significantly imbalanced on a per-topic and inter-topic basis.These make multi-domain stance detection a challenging task, requiring standardization and domain adaptation.To overcome this challenge, we propose Topic Efficient StancE Detection (TESTED), consisting of a topic-guided diversity sampling technique and a contrastive objective that is used for fine-tuning a stance classifier.We evaluate the method on an existing benchmark of 16 datasets with in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics, experiments.The results show that our method outperforms the state-of-the-art with an average of 3.5 F1 points increase in-domain, and is more generalizable with an averaged increase of 10.2 F1 on out-of-domain evaluation while using ≤ 10% of the training data.We show that our sampling technique mitigates both inter-and per-topic class imbalances.Finally, our analysis demonstrates that the contrastive learning objective allows the model a more pronounced segmentation of samples with varying labels.
The BIGAI Offline Speech Translation Systems for IWSLT 2023 Evaluation

This paper describes the BIGAI's submission to IWSLT 2023 Offline Speech Translation task on three language tracks from English to Chinese, German and Japanese.The end-to-end systems are built upon a Wav2Vec2 model for speech recognition and mBART50 models for machine translation.An adapter module is applied to bridge the speech module and the translation module.The CTC loss between speech features and source token sequence is incorporated during training.Experiments show that the systems can generate reasonable translations on three languages.The proposed models achieve BLEU scores of 22.3, 10.7 and 33.0 on tst2023 en→de, en→ja and en→zh TED datasets.It is found that the performance is decreased by a significant margin on complex scenarios like presentations and interviews.
PAL to Lend a Helping Hand: Towards Building an Emotion Adaptive Polite and Empathetic Counseling Conversational Agent

The World Health Organization (WHO) has significantly emphasized the need for mental health care.The social stigma associated with mental illness prevents individuals from addressing their issues and getting assistance.In such a scenario, the relevance of online counseling has increased dramatically.The feelings and attitudes that a client and a counselor express towards each other result in a higher or lower counseling experience.A counselor should be friendly and gain clients' trust to make them share their problems comfortably.Thus, it is essential for the counselor to adequately comprehend the client's emotions and ensure client's welfare, i.e. s/he should adapt and deal with the clients politely and empathetically to provide a pleasant, cordial and personalized experience.Motivated by this, in this work, we attempt to build a novel Polite and empAthetic counseLing conversational agent PAL.To have client's emotion-based polite and empathetic responses, two counseling datasets laying down the counseling support to substance addicts and crime victims are annotated.These annotated datasets are used to build PAL in a reinforcement learning framework.A novel reward function is formulated to ensure correct politeness and empathy preferences as per client's emotions with naturalness and non-repetitiveness in responses.Thorough automatic and human evaluation showcases the usefulness and strength of the designed novel reward function.Our proposed system is scalable and can be easily modified with different modules of preference models as per need 1 .
Distractor Generation for Fill-in-the-Blank Exercises by Question Type

This study addresses the automatic generation of distractors for English fill-in-the-blank exercises in the entrance examinations for Japanese universities.While previous studies applied the same method to all questions, actual entrance examinations have multiple question types that reflect the purpose of the questions.Therefore, we define three types of questions (grammar, function word, and context) and propose a method to generate distractors according to the characteristics of each question type.Experimental results on 500 actual questions show the effectiveness of the proposed method for both automatic and manual evaluation.
Textual Entailment for Temporal Dependency Graph Parsing

We explore temporal dependency graph (TDG) parsing in the clinical domain.We leverage existing annotations on the THYME dataset to semi-automatically construct a TDG corpus.Then we propose a new natural language inference (NLI) approach to TDG parsing, and evaluate it both on general domain TDGs from wikinews and the newly constructed clinical TDG corpus.We achieve competitive performance on general domain TDGs with a much simpler model than prior work.On the clinical TDGs, our method establishes the first result of TDG parsing on clinical data with 0.79/0.88micro/macro F1.Our code is available at https://github.com/Jryao/thyme_tdg.